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ABSTRACT 


Introducing  simulation  training  to  ground  officers  for  the  first  time,  within  an 
existing  proven  curriculum,  presents  a  number  of  challenges  and  questions.  The  proper 
amount  of  simulation  time  to  evaluate  and  train  skill  sets  and  how  to  introduce  simulation 
into  an  existing  curriculum  are  mostly  unknown.  We  have  completed  two  studies  at  the 
Naval  Postgraduate  School  (NPS).  The  first  study  examined  the  length  of  time  and  the 
most  appropriate  method  for  introducing  simulation  training  to  a  user.  The  second  study 
compared  the  use  of  the  Close  Combat  Marines  (CCM)  Tactical  Decision-Making 
Simulation  (TDS)  with  the  traditional  method  of  training  decision-making  called  the 
Tactical  Decision-Making  Game  (TDG).  The  TDS  and  TDG  were  used  in  a  between- 
subjects  experimental  design  to  examine  the  viability  of  each  with  regard  to  their  ability 
to  evaluate  several  important  military  traits.  We  found  that  both  the  TDG  and  the  TDS 
methods  were  useful  in  evaluating  a  participant’s  leadership  characteristics  and  decision¬ 
making  ability.  However,  only  the  TDS  was  capable  of  evaluating  situational-awareness. 
Our  results  also  address  a  novel  way  in  which  these  two  approaches  could  be  combined 
to  amplify  each  other’s  potential  in  training  of  ground  officers  and  military  personnel  in 
general. 
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I. 


INTRODUCTION 


A.  PROBLEM  STATEMENT 

Traditional  training  of  military  ground  officers  in  tactics  has  evolved  over  the 
decades  into  a  proven  crawl,  walk,  run  approach.  Students  are  first  taught  the  tactics, 
techniques,  and  procedures  (TTPs)  in  a  classroom  that  they  will  later  execute  in  the  field. 
The  second  phase  is  rehearsal  in  a  controlled  environment.  This  is  still  one  step  removed 
from  actual  implementation.  Finally,  after  they  have  mastered  the  TTPs,  they  are 
encouraged  to  apply  this  knowledge  in  a  training  session  that,  most  closely,  approximates 
the  real  world  in  hopes  that  the  jump  from  training  to  real  will  be  as  small  as  possible. 
Training  results  over  time  have  shown  that  this  methodical  approach  is  an  effective  way 
to  learn  complex  skills  and  team  tasks.  However,  military  educators  have  struggled  with 
important  questions  about  introducing  a  new  category  of  computer-based  instruction  into 
an  otherwise  proven  curriculum. 

As  technology  has  increased  over  the  last  20  years,  military  educators  have  begun 
to  incorporate  various  forms  of  simulation  and  computer-based  training  into  the  crawl 
and  walk  phases  of  ground  officer  training.  School  administrators  and  teachers  chose  this 
approach  because  it  made  practical  sense  and  provided  a  safe  environment  in  which  to 
train.  Using  this  particular  form  of  training  system,  students  can  make  mistakes  with  few 
detrimental  consequences.  Some  studies  have  demonstrated  that  simulation  technology 
can  help  students  learn  faster,  acquire  more  skills  during  sessions,  and  retain  the 
knowledge  for  a  longer  period.  Some  types  of  simulation  training  expose  people  to 
training  situations  that  would  otherwise  be  extremely  difficult  to  implement.  Real  world 
training,  such  as  artillery  call-for-fire  or  close  air  support  training,  can  be  very  costly  in 
terms  of  the  number  of  units  involved,  time,  and  amount  of  resources  required.  To  train 
one  officer  in  a  call-for-fire  mission  or  a  close  air  support  task,  with  real  assets,  requires 
artillery  pieces  firing  and/or  aircraft  flying.  In  a  virtual  environment,  it  is  possible  to 
train  many  personnel  in  these  domains  at  one  time  without  the  cost  of  real  supporting 
assets. 
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These  are  only  some  of  the  important  benefits  that  simulation  can  offer  to  the 
military  when  real  world  environments  are  found  to  be  cost  prohibitive.  Simulation 
should  not  be  considered  the  ultimate  replacement  for  real  world  field  exercises  because 
there  are  benefits  that  real  world  training  can  offer  the  student  that  simulation  cannot. 
However,  it  can  bridge  the  gap  left  open  when  real  world  training  cannot  be 
accomplished  for  various  reasons. 

Many  studies  have  explored  simulation  training  relative  to  aviators,  but  few  have 
examined  it  specifically  for  use  among  ground  officers.  The  challenge  is  not  introducing 
it  in  the  first  place,  but  to  identify  the  most  effective  way  to  introduce  and  use  it  in 
training  practice.  Young  officers,  who  are  in  the  early  stages  of  their  training,  might 
benefit  from  a  targeted  application  of  simulation  technology.  These  early  stages  could  be 
the  location  in  the  training  cycle  where  simulation  methods  could  prove  to  be  the  most 
useful. 

This  study  will  focus  on  a  comparative  evaluation  of  a  traditional  method  and  a 
new  type  of  simulation  training.  We  trust  that  these  insights  will  help  answer  questions, 
as  well  as,  offer  guidelines  to  a  military  school  that  is  considering  the  integration  of 
simulation  into  its  training  cycle.  The  answers  to  these  questions  will  doubtlessly 
influence  the  welfare  of  military  trainees  who  will  be  a  part  of  the  real  world 
engagements  soon  after  the  completion  of  their  training  regimens. 

B.  RESEARCH  QUESTIONS 

Here  is  a  list  of  questions  that  provided  much  of  the  motivation  for  our  work. 
These  questions  were  not  only  of  interest  to  us  personally,  but,  also,  to  the  U.S.  Marine 
Corps’  Basic  School  at  Quantico,  Virginia: 

•  How  much  time  is  needed  to  familiarize  a  student  with  the  system  before 
training  can  actually  begin? 

•  Can  a  Tactical  Decision-Making  Simulation  (Close  Combat  Marines 
(CCM))  be  used  to  evaluate  leadership? 

•  Are  there  objective  measures  of  leadership,  decision-making,  and 
situational  awareness  that  can  be  evaluated  with  a  TDG  and  a  TDS? 
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•  What  are  the  training  differences  between  using  a  TDS  and  a  TDG?  Do 
they  train  the  same  skills?  If  so,  what  is  the  amount  of  training  time 
required  for  each? 

•  Do  the  TDG  and  TDS  enhance  the  ability  of  the  user  to  visualize  the  battle 
space? 

•  How  many  sessions  are  necessary  to  obtain  a  particular  level  of 
competency  in  tactical  decision-making? 

•  Should  TDG  and  TDS  training  be  mutually  exclusive  or  should  they  be 
used  in  a  combined  training  approach? 

•  Should  simulation  be  used  in  a  controlled  instructor-led  environment  or 
should  students  be  allowed  to  exclusively  explore  and  learn  on  their  own? 

This  research,  and  the  subsequent  studies,  was  the  result  of  our  desire  to  explore 
the  previous  concepts  and  questions.  We  addressed  a  selected  set  of  research  questions 
and  elaborated  upon  them  in  this  thesis.  The  remaining  research  questions  were 
considered,  but  not  fully  explored  in  the  document. 

C.  SCOPE 

The  scope  of  this  thesis  will  focus  on  the  previous  research  questions  and  the 
evaluation  of  leadership,  decision-making,  and  situational  awareness  by  using  two 
training  methodologies.  We  will  seek  to  provide  answers  to  military  training  commands 
about  the  use  of  tactical  decision-making  simulations  in  their  training  curricula.  We  will 
compare  and  contrast  the  tactical  decision-making  game  (TDG)  and  the  tactical  decision¬ 
making  simulation  (TDS)  to  see  which  of  these  approaches  is  best  able  to  improve  the 
efficiency  and  the  effectiveness  of  training  in  the  military  environment. 

D.  ORGANIZATION  OF  THESIS 

This  thesis  is  organized  in  the  following  chapters: 

•  Chapter  I  -  Introduction.  This  chapter  provides  the  motivation  for  this 
work,  the  research  questions  asked,  and  the  framework  for  pursuing 
answers  to  these  questions. 

•  Chapter  II  -  Background.  This  chapter  is  a  literature  review  of  scientific 
studies  and  experiments  that  have  contributed  to  the  understanding  of 
leadership,  decision-making,  and  situational  awareness.  The  analysis  and 
introduction  of  the  tactical  decision-making  game  and  tactical  decision¬ 
making  simulation  were,  also,  included.  We,  also,  examined  literature  on 
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a  Tactical  Decision-Making  Simulation  (TDS),  called  Close  Combat 
Marines,  which  the  U.S.  Marine  Corps  is  using  in  its  introductory  officer 
and  non-commissioned  officer  training. 

•  Chapter  III  -  Experiments.  We  conducted  two  experiments.  The  analysis 
of  the  data  collected  in  each  was  the  basis  for  all  conclusions  from  this 
work. 

•  Chapter  IV  -  Recommendations  and  Future  Work.  This  chapter  discusses 
lessons  learned  from  both  experiments  and  offers  suggestions  for  future 
experiments  and  studies.  It,  also,  provides  recommendations  for  future 
versions  of  tactical  decision-making  simulations  like  CCM. 
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II.  BACKGROUND 


A.  INTRODUCTION 

There  are  many  skills  and  attributes  that  young  officers  must  acquire  and  cultivate 
to  become  effective  military  leaders.  Leadership,  decision-making  (DM),  and  situational 
awareness  (SA)  are  three  essential  characteristics  of  good  military  leaders.  It  is, 
therefore,  understandable  that  all  military  schools  endeavor  to  train,  cultivate,  and 
evaluate  their  students  on  these  skills.  At  the  Basic  School  in  Quantico,  Virginia,  Marine 
Corps  lieutenants  are  taught  the  basics  of  infantry  combat.  The  Basic  School’s  mission  is 
to,  “Train  and  educate  newly  commissioned  or  appointed  officers  in  the  high  standards  of 
professional  knowledge,  esprit-de-corps,  and  leadership  required  to  prepare  them  for  duty 
as  company  grade  officers  in  the  operating  forces,  with  particular  emphasis  on  the  duties, 
responsibilities,  and  warfighting  skills  required  of  a  rifle  platoon  commander”  (U.S. 
Marine  Corps  The  Basic  School  Website,  2007). 

All  military  officers  must  lead.  For  them  to  lead  effectively,  they  must  be  able  to 
make  timely  decisions  and  clearly  understand  the  situations  unfolding  around  them.  This 
rationale  led  us  to  select  and  conduct  a  study  of  leadership,  decision-making,  and 
situational  awareness  when  Tactical  Decision-Making  Games  (TDGs)  and  Tactical 
Decision-Making  Simulations  (TDSs)  are  used  as  training  methods. 

B.  LITERATURE  REVIEW 

In  this  section,  we  provide  a  review  of  literature  that  served  as  a  basis  and  a 
starting  point  for  our  work.  The  topics  include  leadership,  tactical  decision-making, 
situational  awareness,  tactical  decision-making  game  training  practice,  and  tactical 
decision-making  simulations. 

1.  Leadership 

As  J.M.  Bums  stated  in  his  book  entitled  Leadership,  “Leadership  is  one  of  the 
most  common  and  least  understood  phenomena  in  the  world”  (as  cited  in  Popper,  1996, 
p.  15).  This  subject  has  been  the  center  of  learned  discussion  for  millennia.  Thinkers 


5 


have  debated  its  origin,  nature,  and  the  subtle  differences  between  what  some  call 
leadership  and  others  call  management.  Leadership  is  a  quality  that  many  claim  to  know 
when  they  see,  but  generally  have  difficulty  describing.  Perhaps  this  is  because  the 
subject  is  vast  and  has  so  many  facets  that  require  analysis,  review,  and  research.  The 
nature  of  this  work  will  be  to  focus  on  how  leadership  may  be  measured  and  documented. 
If  needed,  those  measures  could  be  used  as  additional  guidance  when  deciding  what 
specialization  should  be  suggested  to  young  officers.  The  authors  will  examine  only  a 
few  of  the  most  prominent  theories  and  methodologies  for  measuring  leadership,  both  in 
the  real  world,  as  well  as  in  virtual  environments  (VE)s. 

a.  History  of  Leadership  Thought  and  Doctrine 

Early  leadership  theory  can  be  traced  back  to  ancient  civilizations  in 
Egypt,  Babylon,  Eastern  Asia,  and  Greece.  The  6th  century  B.C.  Chinese  writer,  Lao- 
Tzu,  described  the  qualities  of  “the  wise  leader...  as  selfless,  hardworking,  honest,  able  to 
time  the  appropriateness  of  actions,  fair  in  handling  conflict,  and  able  to  empower 
others...  Aristotle  argued  in  Politics  that  leaders  were  to  help  others  seek  virtue;  they 
would  do  so  by  themselves  being  virtuous”  (Antonakis,  Sternberg,  2004,  p.  101).  One 
could  easily  argue  that  good  leadership  has  eternal  characteristics.  If  the  qualities  can  be 
defined,  captured,  and,  measured,  could  they  be  sufficient  to  determine  if  someone  is  a 
leader  and,  if  so,  one  that  is  good  or  bad? 

Leadership  has  had  many  definitions.  J.  Kotter,  writing  the  in  Harvard 
Business  Review,  says  simply  that  leadership  is  “getting  people  to  act  without  coercion” 
(as  cited  in  Popper,  1996,  p.  15).  Internationally  known  professor  and  author,  Henry 
Mintzberg,  believes  that  “the  manager  (like  a  military  commander)  does  many  things: 
coordination,  logistics,  management  of  information,  budgets,  and  so  forth.  One  of  their 
roles  is  leadership:  motivating  people  to  perform  tasks  to  the  best  of  their  ability”  (as 
cited  in  Popper,  1996,  p.  15).  These  two  definitions  hint  at  a  subtle  difference  between 
managers  and  those  who  lead.  There  are  two  competing  philosophies  when  seeking  to 
articulate  the  definition  of  leadership.  Throughout  history,  great  leaders  exhibited  certain 
qualities.  How  they  obtained  these  qualities  is  the  crux  of  this  long-standing  controversy. 


6 


Are  these  traits  innate  or  are  they  learned?  One  belief  is  that  any  person,  who  so  desires, 
may  learn  the  traits  associated  with  effective  leadership  and  implement  them  (“ learned 
trait ”  approach).  If  he  is  able  to  do  so,  then  he  will  be  an  effective  leader.  The  other 
view  is  that  these  traits  are  not  learned,  but  innate.  This  is  the  classic  argument  that 
leaders  are  bom,  not  made.  The  “bom  not  made ”  philosophy  asserts  that  the 
quantification  of  leadership  characteristics  are  for  the  identification  of  leaders,  not  for  the 
training  of  such.  To  measure  the  effectiveness  of  a  leader,  one  must  identify  and 
categorize  those  traits/qualities,  form  a  scale  on  which  to  measure  each  of  them,  evaluate 
them  individually,  and  finally  examine  them  collectively.  However,  an  additional 
element  must  be  present  in  both  cases.  A  person,  who  has  these  qualities,  whether 
learned  or  bom,  must  aspire  to  leadership.  He  must,  also,  want  it.  If  not,  he  will  avoid 
opportunities  to  lead  and  never  realize  his  potential. 

b.  Modem  Military  Leadership  Training 

The  U.S.  military  establishment  ascribes  to  a  learned-trait  approach.  In 
the  military,  and  over  the  last  35  years  of  the  all  volunteer  force,  those  who  have  entered 
service,  especially  in  the  officer  corps,  are  assumed  to  have  this  desire  to  lead  because 
they  are  self-selected.  Desire  is  assumed,  as  well  as  the  notion  that  leadership  can  be 
taught.  The  U.S.  Marine  Corps  Officer  Candidate  School  even  has  this  notion  codified  in 
its  mission  statement:  “...to  educate,  train,  evaluate,  and  screen  officer  candidates  to 
ensure  they  possess  the  moral,  intellectual,  and  physical  qualities  for  commissioning,  and 
the  leadership  potential  to  serve  successfully  as  company  grade  officers  in  the  operating 
forces”  (USMC  Officer  Candidate  School  website).  Terms,  such  as  “educate,”  “train,” 
and  “leadership  potential”  all  imply  that  there  exists  a  capability  to  leam  the  tenants  of 
leadership.  Also,  that  there  is  some  ability  toward  leading  other  people  that  can  be 
acquired.  The  Marines  have  compiled  a  list  of  fourteen  leadership  traits  considered  to  be 
essential  elements  that  leaders  of  all  organizational  levels  are  encouraged  to  acquire  and 
master.  They  are:  Justice,  Judgment,  Dependability,  Initiative,  Decisiveness,  Tact, 
Integrity,  Enthusiasm,  Bearing,  Unselfishness,  Courage,  Knowledge,  Loyalty,  and 
Endurance. 
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Retired  Army  Lieutenant  General  Walter  F.  Ulmer  Jr.  argued,  in  a 
January/February  2006  article  from  Armor  magazine,  that  leadership  is  not  a  collection  of 
“traits,  personality,  ambition,  intellect,  energy,  and  personal  goals,”  but  “in  practical 
terms,  the  process  of  leading  is  best  measured  not  by  attributes  or  characteristics,  but  by 
the  specifics  of  what  leaders  do”  (Ulmer,  2006,  p.  39).  He  goes  on  to  discuss  a  study  of 
four  division  commanders  who  had  just  completed  tours  of  duty  in  Operation  Iraqi 
Freedom  (OIF).  This  study  examined  “behaviors  that  are  crucial  for  contemporary  leader 
effectiveness”  (Ulmer,  2006,  p.  39)  In  addition  to  this  study,  Lieutenant  General  Ulmer 
looked  at  several  other  historical  studies  and  found  that  a  number  of  specific  behaviors 
continued  to  surface  in  leaders  who  were  widely  thought  of  as  good.  He  compiled  a  list 
from  a  variety  of  different  levels  of  leadership.  It  was  refined  by  an  Army  War  College 
(AWC)  group  of  professors  and  students  into  a  list  containing  thirty  items  as  seen  in 
Figure  1.  These  have  been  further  refined  in  Figure  2  into  a  subset  called,  the  “Big  12.” 

Leader  Behavior  Preferences  ( LB P)  Worksheet 


(Behaviors  that  create  a  command  climate  that  supports  operational  excellence 
and  motivates  competent  people  to  continue  military  service.) 


Specific  leader  behavior  (for  a  division  commander) 

1 .  Adapts  quickly  to  new  situations  and  requirements. 

2.  Understands  and  employs  current  Army  and  Joint  doctrine. 

,  3.  Keeps  cool  underpressure. 

4.  Knows  how  and  when  lo  involve  others  in  decisionmaking. 

5.  Clearly  explains  missions,  standards,  and  priorities. 

6.  Sees  the  big  picture;  provides  context  and  perspective. 

7.  Sets  high  slandards  without  a  "zero  defects"  mentality. 

B.  Encourages  initiative  and  welcomes  new  ideas. 

9.  Backs  up  subordinates;  confronts  the  boss  if  necessary. 

10.  Is  trustworthy;  keeps  promises  or  explains  why  he  can’t. 

1 1 .  Employs  units  in  accordance  with  their  capabilities. 

12.  Can  handle  "bad  news” 

13.  Gets  out  of  the  headquarters  and  visits  the  troops. 

14.  Coaches  and  gives  useful  feedback  to  subordinates. 

15.  Sets  a  high  ethical  tone;  demands  honest  reporting. 

16.  Will  share  the  risks  and  hardships  of  his  soldiers. 

;  17.  Knows  how  to  delegate  and  not  "micromanage.” 

18.  Is  consistent  and  predictable  in  his  behavior. 

19.  Shows  respect  and  consideration  for  others  of  any  rank, 

20.  Puts  mission  and  people  ahead  of  his  own  career. 

21.  Is  approachable;  listens  to  questions  and  suggestions. 

22.  Can  make  tough,  sound  decisions  on  time. 

£3.  Shares  the  limelight;  gives  due  credit  to  others. 

24.  Senses  unproductive  policies  and  makes  prompt  adjustments. 

25.  Builds  and  supports  teamwork  within  staff  and  among  units. 

26.  Holds  people  accountable  for  their  actions  and  results. 

£7.  Is  more  interested  In  doing  good  than  looking  good. 

28.  Is  fair;  doesn't  play  favorites  with  units  or  people. 

29.  Is  positive,  encouraging,  and  realistically  optimistic. 

30.  Write  in: 

J _ 

Figure  1. 

Note:  This  list  may  ba  reproduced  with  credit  to  AWC  Study  "Leadership  Lessons  al  Division  Command 
Level-2004.'' 


Figure  1.  Leader  Behavior  Preferences  (LBP)  Worksheet  (From:  Ulmer,  2006) 
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“BIG  12” 

At  the  top  of  the  list:  (In  order  of  question  number) 

#  3.  Keeps  cool  under  pressure. 

#  5.  Clearly  explains  missions,  standards,  and  priorities. 

#  6.  Sees  the  big  picture;  provides  context  and  perspective. 

#  22.  Can  make  tough,  sound  decisions  on  time. 

Also  particularly  significant:  (In  order  of  question  number) 

#  1 .  Adapts  quickly  to  new  situations  and  requirements. 

#  7.  Sets  high  standards  without  a  “zero  defects”  mentality. 

#  1 2.  Can  handle  “bad  news.” 

#  14.  Coaches  and  gives  useful  feedback  to  subordinates. 

#  15.  Sets  a  high  ethical  tone;  demands  honest  reporting. 

#  1 7.  Knows  how  to  delegate  and  not  “micromanage” 

#  25.  Builds  and  supports  teamwork  within  staff  and  among  units. 

#  29.  Is  positive,  encouraging,  and  realistically  optimistic. 

Figure  2 


Figure  2.  The  “Big  12”  (From:  Ulmer,  2006) 


Micha  Popper,  Professor  of  Psychology  of  Haifa  University,  Haifa,  Israel, 
compares  leadership  in  the  military  and  management  in  the  business  world.  Some  might 
argue  that  these  are  two  names  for  the  same  thing.  However,  Popper  demonstrates  that 
there  is  a  qualitative  difference  between  the  perceptions  of  a  leader  in  and  outside  the 
military  context.  In  a  1981  study  examining  leadership  styles  of  military  officers  in  Viet 
Nam,  Gabriel  and  Savage  assert  that  military  leadership  in  Vietnam  was  overly 
influenced  by  a  business-like  sterile  managerial  style  and  not  in  keeping  with  traditional 
military  leadership  principles.  This  over  reliance,  or  commercial  management 
techniques,  led  to  disastrous  consequences  in  Viet  Nam  (as  cited  by  Popper,  1996). 
Popper  describes  two  very  different  relationships  that  exist  between  the  leaders  and  the 
led: 


1.  The  Transactional  Leader  -  The  transactional  leader  is  the  leader 
who  assesses  his  team’s  psychological  needs  and  determines  the  link 
between  their  amount  of  effort  and  reward  for  that  effort.  He  is 
“transactional”  in  the  sense  that  he  is  a  broker  of  tangible  and  intangible 
incentives  that  are  used  to  persuade  his  followers  to  accomplish  some 
goal.  This  is  most  associated  with  a  successful  businessperson  who 
orchestrates  a  connection  between  corporate  goals  and  team  performance 
(Popper,  1996). 

2.  The  Emotional  Leader  -  The  emotional  leader  is  quite  different, 
though  he  may  still  have  a  transactional  element  to  his  nature.  He  leads 
primarily  by  eliciting  positive  emotional  responses  from  his  followers. 
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The  emotional  leader  is  one  who  “may  be  described  by  images,  such  as 
charismatic,  visionary,  and  inspirational”  (Popper,  1996,  p.  16).  The 
emotional  leader  is  generally  more  common  in  military  environments 
where  tangible  incentives  are  less  common  and  intangibles  dominate. 
This  is  the  type  of  leader  who  is  capable  of  arousing  emotions  so  strong 
that  “people  are  even  willing  to  sacrifice  their  lives  for  the  leader  (Popper, 
1996,  p.  16). 


c.  Leadership  Studies 

(1)  Measuring  Military  Development  (MD).  In  the  opening 
chapter  of  his  book,  The  West  Point  Way  of  Leadership,  retired  Army  Colonel  Larry  R. 
Donnithome  describes  West  Point’s  philosophy  of  leadership: 

At  every  Fortune  500  institution  in  America,  people  are  taught  ethics.  At 
West  Point,  people  are  taught  character.. .A  leader  of  character  has  all  the 
qualities  we  normally  associate  with  leaders’  ambition,  confidence, 
courage,  intelligence,  eloquence,  responsibility,  creativity,  compassion 
and  one  thing  more  which  we  unfortunately  overlook  too  frequently 
among  civilian  leaders:  A  leader  of  character  is  absolutely  trustworthy, 
even  in  times  of  great  stress,  and  can  be  depended  upon  to  put  the  needs  of 
others,  the  organization,  the  community  above  personal  considerations, 
not  now  and  then,  or  when  the  spirit  moves  him,  or  when  it  will  look  good 
on  his  resume,  but  in  every  instance  (Donnithome,  1994). 

Paul  Bartone,  Scott  Snook,  and  Trueman  Tremble  (2002), 
embarked  on  a  4-year  longitudinal  study  of  West  Point  cadets  (N=1143)  that  examined 
the  influence  of  cognitive  and  personality  variables  on  military  leadership  performance. 
Due  to  attrition  of  some  25%  of  the  cadet  class  over  the  course  of  four  years,  the  final 
number  of  cadets  in  the  study  was  N=855.  A  cross-validated  hierarchical  multiple 
regression  procedure  was  used  to  determine  the  factors  upon  entry  that  successfully 
predicted  military  development  grades  in  upper-classmen.  The  following  potential  traits 
were  identified  as  potential  predictors  of  leader  performance: 

•  Spatial  Judgment 

•  Logical  Reasoning 

•  Social  Judgment 

•  Problem  Solving 

•  College  Entrance  Equivalency  Rating  (CEER) 
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In  addition  to  the  previous  five  predictors  of  leadership 
performance,  the  researchers,  also,  examined  personality  based  on  the  Neuroticism, 
Extraversion,  Openness,  Agreeableness,  and  Conscientiousness  (NEOAC)  personality 
inventory  (Costa  &  McCrae,  1985).  The  process  used  to  evaluate  these  predictors  was  to 
first  examine  the  potential  affects  of  the  varying  demographic  variables  (sex,  race,  age) 
on  upperclassman  leader  performance  scores.  Then,  move  further  into  analyzing  the 
cognitive  and  personality  variables.  If  correlations  were  not  established,  then  that 
particular  variable  was  dropped  from  further  consideration  (Bartone,  Snook,  Tremble, 
2002). 

The  researchers  examined  demographic,  cognitive,  and  personality 
variables  with  the  following  results:  There  was  no  correlation  found  between  the 
demographic  variables  of  race  and  age  with  leader  performance  scores;  thus,  these  were 
dropped.  There  was  a  correlation  with  sex;  thus,  this  variable  was  retained  for  further 
consideration.  Of  the  cognitive  variables  considered  in  the  study,  problem  solving  and 
spatial  judgment  (mental  figure  rotation)  did  not  correlate  with  leader  performance 
scores.  These  were,  also,  dropped  from  further  consideration.  Logical  reasoning,  social 
judgment,  and  CEER  did  correlate  significantly  with  leader  performance  and  were  kept 
for  additional  analysis.  Of  the  personality  variables,  neuroticism  and  openness  did  not 
correlate  with  leader  performance  scores  and  were  not  analyzed  further.  Extraversion, 
agreeableness,  and  conscientiousness  did  correlate  with  leader  performance;  thus,  they 
were  kept  (Bartone,  Snook,  Tremble,  2002).  After  the  preliminary  correlations,  the 
following  variables  remained  for  consideration:  sex,  logical  reasoning,  social  judgment, 
CEER  score,  extroversion,  agreeableness,  and  conscientiousness.  Based  on  a  multiple 
regression  analysis,  the  authors  state  that  the  previously  mentioned  demographic, 
cognitive,  and  personality  variables  must  be  considered  predictive  of  leadership 
performance  over  time.  After  two  hierarchical  multiple  regression  analyses  were  done  on 
these  data,  all  variables  were  eliminated,  leaving  sex,  college  entrance  scores,  social 
judgment,  and  conscientiousness  as  predictors  of  leadership  performance. 
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(2)  Popper’s  Big  Three  and  Their  Subcategories.  Professor 
Micha  Popper  asserts  that  there  are  three  types  of  psychological  capacities  that  are 
essential  for  leadership:  self-confidence,  a  proactive  orientation,  and  a  capacity  for  pro¬ 
social  relationships.  The  three  capacities  cited  here  by  Dr.  Popper  are  based  on  the  initial 
research  of  professor  and  author  Bernard  Bass  (Bass,  1990).  In  the  study  published  in 
Military  Psychology  in  2004,  Popper  et  al.  administered  questionnaires  to  402  Israeli 
Defense  Force  (IDF)  soldiers  from  infantry  and  armor  corps  units  as  they  approached  the 
end  of  their  basic  training.  These  were  compared  with  a  socio-metric  questionnaire 
examining  the  peer  and  commander  evaluations  of  the  soldiers’  leadership  capacities 
which,  in-tum,  were  used  to  evaluate  each  soldier  and  to  classify  the  soldiers  either  as 
leaders  or  non-leaders.  Findings  from  this  study  revealed  a  significant  difference 
between  leaders  and  non-leaders  in  all  variables  defined  as  psychological  capacities  to 
lead  (Popper,  2004).  The  researchers  evaluated  participants  on  the  following  levels: 

1.  Locus  of  Control  (LOC)  -  refers  to  the  capacity  that  one  believes 
he  has  to  control  events  in  his  life. 

2.  Self-Confidence  -  consists  of  internal  locus  of  control,  low  level  of 
trait  anxiety,  and  self-efficacy. 

3.  Trait  Anxiety  -  is  characterized  by  the  degree  to  which  a  person 
will  either  exhibit,  or  not  exhibit,  symptoms  of  stress  when  faced 
with  a  variety  of  situations. 

4.  Self-efficacy  -  is  defined  as  a  person’s  belief  that  he  can  face  and 
accomplish  a  task  or  perform  successfully  when  presented  with  a 
goal. 

5.  Proactive  Orientation  -  is  measured  by  level  of  optimism. 

6.  Capacities  Required  for  Pro-Social  Relationships  -  is  a  measure  of 
how  well  the  person  gets  along  with  others. 

Popper’s  findings  resulted  in  the  following  conclusions.  From  the 
above  six  measures,  leaders  were  found  to  exhibit: 

1.  Higher  levels  of  internal  locus  of  control  when  compared  to  non¬ 
leaders. 

2.  Lower  levels  of  anxiety  compared  with  non-leaders 

3.  Higher  levels  of  self-efficacy  compared  with  non-leaders. 

4.  Higher  levels  of  optimism  compared  with  non-leaders. 
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5.  Higher  levels  of  secure  attachment  style  compared  with  non 
leaders. 

6.  Lower  avoidant  and  anxious  attachment  styles  compared  with  non¬ 
leaders  (Popper,  2004,  p.  257). 

d.  Measuring  Leadership  in  a  Virtual  Environment  (VE) 

Leadership  in  a  virtual  environment  is  not  that  far  removed  from 
leadership  in  real  environments.  Many  of  the  characteristics  for  evaluating  leadership 
potential  are  essentially  the  same.  However,  research  conducted  by  Steed,  et  al.  in  1999, 
a  group  of  researchers  specializing  in  real  time  virtual  environments  from  the  University 
College  in  London,  demonstrated  that,  in  some  cases,  immersion  within  a  virtual 
environment  was  positively  correlated  with  leader  behavior. 

In  this  paper,  researchers  discussed  a  series  of  three  studies  that 
investigated  small  group  collaboration  within  a  virtual  environment.  Groups  consisted  of 
three  individuals  who  were  tasked  with  solving  a  task  that  required  a  high  degree  of 
collaboration  among  the  participants.  The  participants  manipulated  avatars  that  were 
colored  red,  green,  and  blue.  They  initially  met  in  a  virtual  room  and,  then,  were  tasked 
with  solving  a  series  of  puzzles  that  consisted  of  fragmentary  statements  that  were  written 
on  pieces  of  paper  hung  on  the  walls  in  different  parts  of  the  virtual  environment.  The 
participants  were  required  to  locate  all  pieces  of  paper,  and  rearrange  the  phrases  on  each 
individual  sheet  to  form  an  intelligible  sentence.  After  the  sessions,  the  researchers 
examined  the  relationships  between  the  resources,  presence,  co-presence,  immersion, 
group  accord,  and  leadership. 

Study  1  -  The  first  experiment  was  conducted  with  all  participants  located 
at  the  same  university.  In  this  study,  two  of  the  participants  worked  on  desktop 
computers  and  the  third  participant  was  immersed  via  a  head-mounted  display  (HMD). 
They  started  the  task  in  a  virtual  environment  and  finished  it  in  a  real  environment. 
There  were  ten  groups  of  three  that  went  through  this  study.  The  results  suggested  that 
the  person  who  was  immersed  with  the  HMD  tended  to  emerge  as  the  leader  for  a 
significant  amount  of  time.  Additionally,  group  accord  tended  to  be  higher  in  the  real 
meeting  than  in  the  virtual  meeting. 
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Study  2  -  The  second  experiment  recreated  the  first  study,  but  conducted 
the  study  over  a  wide  area  network.  In  this  study,  no  real  environment  was  present.  All 
participants  were  placed  in  and  remained  in  the  virtual  environment.  No  one  participant 
was  immersed  more  than  the  others  and  no  participant  emerged  as  a  leader.  Group 
accord  matched  the  results  of  Study  1 . 

Study  3  -  In  the  final  experiment,  the  two  participants  were  physically 
located  in  London  and  Nottingham,  England,  and  one  in  Greece.  One  participant  was 
immersed  more  than  the  others  via  HMD.  This  was  required  to  study  whether  use  of  a 
wide  area  network  would  have  any  effect  on  immersion.  Dependent  variables  were  self- 
reported  presence  as  indicated  via  a  questionnaire.  Co-presence  was,  also,  assessed  via 
questionnaire.  Group  accord  was  measured  similarly  with  a  questionnaire.  To  account 
for  a  confound  associated  with  personality,  another  question  was  added  to  control  for 
anxiety  effects  resulting  from  the  study. 

The  results  from  a  statistical  analysis  of  the  surveys  found  a  positive 
correlation  between  presence  and  co-presence.  Immersion  had  little  or  no  effect  at  all  on 
reported  presence.  Individual  accord  was  positively  correlated  with  co-presence.  No 
clear  pattern  of  leadership  behavior  emerged  with  respect  to  immersion.  However,  it  was 
noted  that  the  green  participant  (Greece)  almost  never  emerged  as  the  leader.  This  was 
likely  a  result  of  the  wide  area  connection  being  3-5  times  slower  than  the  network  speed 
between  London  and  Nottingham.  Researchers,  also,  found  that  the  immersed  person 
with  the  HMD  tended  to  emerge  as  the  leader  a  significant  amount  of  time.  Steed,  et  al., 
hypothesized  that  this  was  the  case  because  he  was  presented  with  superior  navigation 
metaphors  (walking  and  moving  naturally  in  space)  and,  therefore,  he  managed  to  reach 
more  information  about  the  environment  than  the  others.  That  person,  perhaps  for  the 
same  reason,  felt  compelled  to  be  more  vocal  about  his  findings  and  felt  obligated  to 
provide  his  teammates  with  the  best  information  possible.  Consequently,  the  same  person 
was  perceived  as  someone  who  knew  what  he  was  doing  i.e.,  someone  who  was  more 
likely  to  be  perceived  as  a  leader  (talkativeness  did  correlate  highly  and  positively  with 
leadership). 
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e.  Summary 

Many  of  the  studies  that  we  mentioned  in  this  section  found  a  number  of 
factors  believed  to  predict  or,  at  least,  to  correlate  with  leadership.  For  instance,  the  U.S. 
Marine  Corps  publishes  their  fourteen  leadership  traits.  Further,  Lieutenant  General 
Ulmer’s  article  in  Armor  magazine  distills  the  leadership  behaviors  from  four  division 
commanders  with  combat  experience  in  Iraq  down  to  a  list  of  thirty  behaviors  (Figure  1). 
Also,  in  Lieutenant  Colonel  Donnithome’s  book,  The  West  Point  Way  of  Leadership,  he 
mentions  nine  leadership  characteristics.  Bartone,  et  al.,  and  Popper  established  a 
number  of  predictive  leadership  measures. 

A  simple  comparison  yielded  an  interesting  juxtaposition  of 
traits/behaviors  in  Table  1  from  Lao-tzu  to  modem  military  perspectives.  The  idea 
behind  this  table  is  to  create  a  quick  mechanism  to  compare  each  of  the  studies/lists 
mentioned  in  this  work  to  identify  if  there  is  agreement  between  the  different 
methodologies  to  identify  leadership  qualities  (for  either  identification  or  predictive 
purposes).  After  the  chart  was  established,  we  removed  rows  containing  only  one  item. 
The  purpose  of  this  was  to  demonstrate  agreement  between  the  methods  —  not  to  list  each 
line  item  represented. 
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Table  1.  Comparative  Analysis  of  Previously  Mentioned  Observations  &  Research 


Confident  and  Calm  Under  Pressure 

♦ 

♦ 

♦ 

♦ 

Courageous 

♦ 

♦ 

Decisive 

♦ 

♦ 

♦ 

Eloquent  and  Articulate 

♦ 

♦ 

Encourages  Initiative 

♦ 

♦ 

♦ 

Fair 

♦ 

♦ 

♦ 

♦ 

Intelligent 

♦ 

♦ 

Positive  and  Enthusiastic 

♦ 

♦ 

♦ 

Professionally  Competent 

♦ 

♦ 

Selfless  and  Loyal 

♦ 

♦ 

♦ 

Sound  Judgment 

♦ 

♦ 

♦ 

♦ 

Trustworthy 

♦ 

♦ 

♦ 

♦ 

Most  of  these  are  a  very  close  match,  but  there  was  some  subjectivity  in 
the  placement  and  assignment  of  each  leadership  trait.  Intelligence  was  not  specifically 
mentioned  by  Ulmer,  nor  was  it  listed  among  the  USMC  leadership  traits,  but,  from  the 
other  behaviors  and  traits  that  were  listed,  it  becomes  obvious  that  this  was  an  important 
element  of  the  leadership  equation.  Perhaps,  this  might  be  one  that  the  authors  believed 
was  innate  and  individualistic;  thus,  it  could  not  be  acquired  or  improved  upon  as  the 
other  traits  listed  in  the  chart.  These  findings  are  interesting  for  a  number  of  reasons  and 
have  obvious  implications  for  military  training.  The  NEOAC  is  such  a  simple  instrument 
which  could  be  used  during  initial  military  training  to  establish  baselines  of  leadership 
potential.  Virtual  environments  might  be  the  next  step  in  the  evaluation  of  a  student  with 
respect  to  his  leadership  grade.  This  assumes  that  the  VE  was  constructed  to  create 
situations  where  leadership  could  be  demonstrated.  We  believe  that  additional  focused 
studies  should  be  conducted  to  confirm  if  leadership  can  be  predicted  based  on  specific 
observations  of  a  user’s  participation  in  a  VE  (Steed  et  al.,  1999).  Conclusive  evidence  in 
this  regard  could  provide  the  instructors  with  early  indicators  of  how  a  student  might 
perform  in  a  real  leadership  situation.  Though  researchers  are  still  unable  to  write  a 
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definitive  prescription  for  leadership,  they  are,  at  least,  closer  to  that  goal  than  Lao-tzu 
and  Aristotle  were  two  to  three  millennia  previously.  Interestingly,  the  principles  that 
these  two  philosophers  saw  so  long  ago  in  the  leaders  of  their  day  are  still  both  relevant 
and,  in  some  cases,  the  same  ones  that  philosophers  today  use  to  measure  leadership. 

2.  Decision-Making 

Decision-making  is  an  essential  part  of  military  operations.  Modem  scientific 
research  has  developed  numerous  models  of  decision-making.  Military  organizations 
have  incorporated  some  of  these  elements  into  their  processes  as  well.  The  Military 
Decision-Making  Process  (MDMP)  has  been  documented  to  assist  military  personnel 
with  organizing  their  thoughts  and  preparing  courses  of  action.  Most  recently,  military 
schools  have  adopted  computer-based  decision-making  simulations  as  a  way  to  train 
small  unit  leaders  in  the  art  of  tactical  decision-making. 

a.  The  History  of  Military  Decision-Making  Doctrine 

Militaries  throughout  history  have  relied  upon  good  decision-making 
methodologies  to  implement  their  tactics  and  strategies.  The  U.S.  Army  Field  Manual 
(FM)  100-5  Operations  defines  decision-making  as  a  combination  of  information  coupled 
with  use  of  judgment  as  an  element  of  combat  power.  It  emphasizes  that  decision¬ 
making  requires  knowing  the  following  elements:  If  to  decide,  when  to  decide,  and  what 
to  decide  (as  cited  in  Beal,  2002,  p.  1).  A  method  of  decision-making  is  a  critical  skill 
from  the  highest  echelons  within  the  military  establishment  to  the  level  of  the  individual 
soldier.  To  make  quick  course  of  action  decisions,  the  leader  must  practice  making 
decisions  based  on  a  consistent,  logical,  and  deliberate  process. 

Carl  von  Clausewitz,  the  famous  Prussian  military  strategist  of  the  early 
19th  century,  compares  decision-making  at  the  tactical  and  strategic  levels: 

It  takes  much  greater  strength  of  will  required  to  make  key  strategic 
decisions  than  a  tactical  one.  With  tactics,  the  actor  is  swept  along  by  the 
moment  and  feels  caught  up  in  a  whirlwind  so  intense  that  the  struggle 
against  it  would  result  in  the  direst  of  consequences.  With  strategy,  where 
everything  moves  much  more  slowly,  there  is  plenty  of  room  for  one’s 
own  misgivings,  objections,  and  ideas  -  and  those  of  others  -  and  for 
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inopportune  remorse.  With  strategy,  one  does  not  see  at  least  half  the 
situation  with  one’s  own  eyes;  rather,  everything  must  be  guessed  at  and 
presumed,  which  decreases  one’s  level  of  conviction.  As  a  result,  most 
generals  become  bogged  down  with  ineffectual  fears  when  they  should  be 
taking  action,  (as  cited  in  Ghyczy,  von  Oetinger,  Bassford,  2001). 

Clausewitz  clearly  states  that  the  tactical  leader  is  required  to  make 
decisions  faster  than  a  leader  at  the  strategic  level.  Decisions  at  the  tactical  level  usually 
involve  close  to  immediate  life  or  death  situations.  For  the  leader  at  the  tactical  level  to 
practice  his  decision-making,  he  must  be  in  an  environment  where  he  must  make 
decisions  and  experiences  their  consequences.  Tactical  decision-making  simulations 
provide  both  the  environment  and  the  consequences  for  today’s  military  leaders. 

b.  Decision-Making  in  Today’s  Military 

Decision-making  is  not  only  a  concept  written  about  in  the  historical 
strategy  books,  but  one  that  is  crucial  for  victory.  Military  field  manuals,  related  to 
strategy  and  tactics  on  the  battlefield,  require  good  decision-making  processes.  Chapter  2 
of  FM  17-15  (1996),  describes  the  concept  of  military  command  as  having  two  vital 
components:  decision-making  and  leadership.  Furthermore,  the  same  manual  describes 
decision-making  as  a  conscious  process  for  selecting  a  course  of  action  from  two  or  more 
alternatives  (FM  17-15,  1996).  The  importance  of  decision-making  is,  also,  clearly  seen 
in  infantry  platoon-level  field  manuals.  FM  7-8,  Infantry  Rifle  Platoon  and  Squad,  states 
that  decision-making  in  one  of  the  most  important  skills  that  a  leader  can  have. 
Furthermore,  it  states  that  the  platoon  and  squad  leaders  should  be  tacticians  and  take  the 
initiative  on  the  battlefield.  For  them  to  make  sound  decisions,  they  must  master 
knowledge  of  tactics,  techniques,  and  procedures  (TTPs)  at  the  squad  and  platoon  levels 
(FM  7-8,  2001,  Chapter  1). 

The  Military  Decision-Making  Process  (MDMP)  is  an  expedited  logical 
set  of  guidelines  that  is  used  when  there  is  more  time  to  develop  and  plan  for  a  course  of 
action.  MDMP  is  a  method  that  can  help  commanders,  and  their  staffs,  to  reach  logical 
decisions  in  a  timely  manner.  The  commander,  and  the  staff,  determines  different 
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courses  of  action  for  both  friendly  and  enemy  forces.  After  this  phase,  they  develop  their 
plan.  The  steps  of  the  MDMP  are  contained  in  Army  FM  101-5,  Chapter  5,  p.  3: 

1 .  Receipt  of  Mission 

2.  Mission  Analysis 

3.  Course  Of  Action  Development 

4.  Course  Of  Action  Analysis 

5.  Course  Of  Action  Comparison 

6.  Course  Of  Action  Approval 

7.  Orders  Production 

The  MDMP  is,  also,  addressed  in  detail  in  FM  5-0,  Staff  Organization,  and 
Operations.  It  describes  the  deliberate  form  of  MDMP  as  both  a  prescriptive  process  and 
an  analytical  tool  (as  cited  in  Van  Poppel,  2005).  The  U.S.  Marine  Corps  has  a  doctrinal 
publication  (Marine  Corps  Warfighting  Publication  5-1),  named  “Marine  Corps  Planning 
Process,  that  approximates  the  Army  method  contained  in  FM  101-5. 

c.  Decision-Making  Models 

There  are  many  types  of  decision-making  models  within  civilian  and 
military  applications.  Military  field  manuals  and  civilian  researchers  perceive  decision¬ 
making  very  similarly.  They  both  understand  it  to  be  the  process  of  choosing  the  most 
appropriate  course  of  action  among  many.  The  two  models  that  are  most  relevant  to  this 
work  are  the  rational  model  and  naturalistic  decision-making. 

Rational  Decision-Making:  Oxford  Business  School  Professor  Loizos 
Heracleous  (1994)  explains  rational  decision-making  as  choosing  the  objective  among  a 
variety  of  alternatives.  Heracleous  also  concludes  that  the  applicability  of  this  model  is 
limited  to  relatively  simple  problems  where  objectives  are  clear,  unambiguous,  agreed 
upon,  and  cause-effect  relations  are  well  understood.  This  definition  of  decision-making, 
and  the  explanation  of  the  rational  model,  is  virtually  identical  to  the  definition  stated  in 
U.S.  Army  FM  101-5.  Baxter  et  al.  (2004)  examined  military  decision-making  and  found 
that  it  was  one  of  the  eight  dimensions  of  tactical  thinking.  Decision-making  is  the 
identification  of  a  feasible  course  of  action  (CO A)  obtained  from  experience  accumulated 
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in  similar  situations.  Christopher  Wickens  et  al.  (2004)  consider  rational  decision¬ 
making  to  be  represented  by  three  phases: 

1 .  Acquiring  and  perceiving  information  cues  relevant  for  decision, 

2.  Generating  and  selecting  hypotheses,  or  situation  assessments, 
about  the  meaning  of  the  cues  regarding  the  current  state  relevant 
to  the  decision, 

3.  Planning  and  selecting  the  choices  to  take,  on  the  basis  of  the 
inferred  state,  the  costs,  and  the  different  outcomes.  The  three 
stages  often  cycle  and  iterate  in  a  single  decision. 

Naturalistic  Decision-Making:  Researchers  soon  discovered  that  humans 
often  did  not  follow  rational  models  for  decision-making  tasks.  Instead,  Gary  Klein’s 
research  on  naturalistic  decision-making  in  1993,  proposed  that  people  use  heuristic  and 
individual  bias  from  previous  similar  experiences  to  make  decisions  (as  cited  in  Shattuck 
&  Miller,  2006).  Naturalistic  decision-making  seeks  to  describe  how  humans  actually 
make  decisions  in  the  environment  where  the  decisions  would  actually  be  made. 
Laboratory  research  of  decision-making  was  considered  artificial  and  ultimately  an 
erroneous  way  to  examine  the  real  way  that  people  make  decisions.  Furthermore, 
Christopher  Wickens  et  al.  (2004)  state  that  people  try  to  make  the  best  possible  decision 
for  their  given  circumstances  and  within  time  limits.  Instead  of  waiting  for  the  very  best 
solution  to  percolate  to  the  top,  they  try  to  find  the  choice  that  is  “good  enough”  for  their 
purposes.  This  shortcut  method  is  termed  as  “satisficing.” 

When  we  look  at  the  military  decision-making,  we  see  that  field  manuals 
do  not  only  propose  the  detailed  and  time  consuming  process  to  arrive  at  a  good  decision 
(MDMP),  but,  also,  they  promote  a  method  of  heuristics  to  evaluate  the  situation 
composed  of  the  following  elements:  Mission,  Enemy,  Terrain,  Troops,  and  Time 
(METT-T).  METT-T  permits  a  commander  to  obtain  a  relatively  comprehensive  mental 
picture  of  his  particular  situation  without  going  through  the  extensive  and  time- 
consuming  MDMP.  The  Marine  Corps  captures  this  particular  concept  in  Marine  Corps 
Doctrinal  Publication  1-3  (Tactics)  which  states: 
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In  some  cases,  speeding  up  the  analytical  decision-making  process  may  be 
sufficient;  however,  in  most  cases,  intuitive  decision-making  is  needed  to 
generate  and  maintain  tempo.  Intuitive  decision-making  relies  on  a 
commander’s  intuitive  ability  to  recognize  the  key  element  of  a  particular 
problem  and  arrive  at  the  proper  decision  without  having  to  compare 
multiple  options  (as  cited  in  Nichols,  p.  26). 

d.  Decision-Making  and  Simulation 

Military  organizations  recognize  that  computer-based  decision-making 
tools  are  important  in  the  training  of  their  personnel.  One  of  the  difficulties  in  acquiring 
good  decision-making  skills  is  that  a  leader  needs  to  make  many  critical  decisions. 
However,  there  may  be  few  of  these  opportunities.  For  example,  the  opportunities  to 
make  tactical  decisions  in  combat  do  not  occur  that  often.  The  best  decision-making 
environment  that  can  be  achieved  in  training  is  one  that  approximates  the  combat 
situation  as  closely  as  possible.  Even  training  exercises  are  limited  due  to  time,  space, 
financial  considerations,  or  physical  limitations  to  conduct  a  large  number  of  trials/runs 
in  a  given  (usually  short)  period  of  time.  One  way  of  providing  a  decision  maker  with  the 
opportunity  to  make  many  decisions  is  to  introduce  the  use  of  tactical  decision-making 
simulations  (TDSs)  in  his  regular  training  schedule.  This  can  place  a  leader  at  a  critical 
decision  point  many  times.  This  method  allows  the  leader  not  only  to  make  decisions 
regarding  how  he  would  proceed  with  his  own  troops,  but,  also,  to  allow  him  to  view  the 
same  situation  from  the  enemy’s  perspective. 

According  to  retired  U.S.  Marine  Corps  Gunnery  Sergeant,  Paul  Nichols 
(2006),  the  focus  is  on  enhancing  Marines’  analytical  decision-making  abilities.  There 
have  been  several  studies  in  which  scientists  evaluated  the  training  effectiveness  of  a 
single  TDS.  In  their  study,  Baxter  et  al  (2004)  tried  to  evaluate  how  effective  a  TDS, 
called  Close  Combat  Marines  (CCM),  was  for  the  subjects  compared  to  traditional  paper- 
based  tactical  decision-making  games  (TDG).  They  did  this  study  at  the  USMC  Infantry 
Platoon  Sergeant  (IPS)  Course,  Advanced  Training  School,  Camp  Geiger,  North 
Carolina.  Fourteen  USMC  non-commissioned  officers,  who  were  attending  the  IPS 
course,  participated  in  their  study.  The  researchers  administered  surveys  to  the  students 
and  instructors.  They  required  the  students  to  choose  a  course  of  action  (COA).  Based 
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on  the  results  of  their  data  analysis,  they  concluded  that  the  TDS  (CCM)  was  most  likely 
to  address  the  macro-cognitive  activities  of  uncertainty  management  and  team 
coordination.  The  TDG  better  addressed  mental  simulation  and  planning.  They  could 
not  find  any  significant  evidence  whether  TDG  or  TDS  was  effective  for  training 
decision-making. 

There  have  been  other  studies  that  evaluated  TDSs  as  well.  In  the  Tactical 
Decision-Making  Simulations  II  report,  which  was  prepared  by  CHI  Systems  and  Klein 
Associates  for  NAVAIR  Orlando  in  2004,  the  researchers  tried  to  evaluate  another  TDS, 
named  Marine  Air  Ground  Task  Force  XXI  (MAGTF  XXI).  The  researchers  stated  that 
situational  awareness  is  the  foundation  for  making  good  decisions.  In  the  beginning  of 
the  NAVAIR  report,  they  hypothesized  that  SA  accuracy  scores  would  be  higher  in  the 
TDG  condition  than  in  the  TDS  condition.  This  was  confirmed  by  their  findings.  They, 
also,  stated  that  follow-on  research  could  investigate  the  best  use  of  TDSs  for  producing 
improvements  in  SA  and  in  assessing  the  impact  of  this  training  on  decision-making. 

In  their  Naval  Postgraduate  School  thesis,  Games  for  Training:  Leveraging 
Commercial  Off  The  Shelf  Multi-player  Gaming  Software  for  Infantry  Squad  Collective 
Training,  Nolan  and  Jones  (2005)  sought  to  find  out  whether  the  first  person  shooter 
games  could  effectively  train  squads  in  collective,  leadership  tasks,  and  decision-making. 
They  chose  “combat  drill”  as  the  task  that  the  experiment  subjects  would  perform 
because  it  is  a  “collective  action  rapidly  executed  without  applying  a  deliberate  decision¬ 
making  process”  (as  defined  in  FM  7-8,  2001,  Chapter  4,  p.  1).  They  concluded  that  the 
infantry  squads  can  use  commercial  off  the  shelf  (COTs)  gaming  software  for  this  level 
and  type  of  collective  squad  training. 

Military  doctrine  and  manuals  emphasize  the  importance  of  leadership  and 
decision-making.  Because  they  believe  they  are  beneficial,  military  schools  have  adopted 
computer-based  methods  for  training  their  young  Commissioned  Officers  and  Non- 
Commissioned  Officers  (NCOs).  Modem  scientific  research  confirms  this  notion  and 
presents  its  own  evidence  that  simulations  have  a  strong  role  to  play  in  this  training 
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effort.  TDSs  offer  the  student  the  ability  to  act  as  if  he  was  leading  a  small  unit  in  a  real 
environment  and  present  him  with  numerous  opportunities  to  make  decisions  in  virtual 
combat. 


3.  Situational  Awareness  (SA) 

Situational  awareness  is  a  concept  that  has  been  the  center  of  much  academic 
discussion.  To  some  degree,  this  is  a  debate  of  semantics  because  most  researchers 
acknowledge  that  humans  have  some  level  of  perception  regarding  the  events  that  are 
taking  place  around  them  and  that  they  use  this  information  to  make  their  decisions.  The 
concept  of  how  SA  is  defined  and  measured  in  teams  and  in  individuals  is  not  only 
debated  in  civilian  circles,  but  within  military  training  environments  as  well.  The 
remainder  of  this  section  will  be  devoted  to  the  discussion  of  concepts  that  are  related  to 
situational  awareness. 


a.  Definitions  of  Situational  Awareness 

Although  there  have  been  many  definitions  of  SA,  Dr.  Mica  Endsley’s 
definition  and  her  model  are  some  of  the  most  well  known  and  widely  accepted  in  the 
field.  Endsley  defines  SA  as  “knowing  what  is  going  on  around  you”  and  “consisting  of 
three  levels:  perceiving  elements  in  the  environment  within  a  volume  of  space  and  time; 
comprehending  what  they  mean  in  context;  and  predicting  their  status  in  the  near  future” 
(as  cited  in  Stanners  &  French,  2005,  p.  2).  In  a  technical  paper  written  for  the  society  of 
automotive  engineers,  W.L.  Hamilton  (1987)  provides  an  earlier  definition  of  SA  in  the 
military  environment  as  the  knowledge  of  current  and  near-term  disposition  of  both 
friendly  and  enemy  forces.  The  authors  of  a  1988  technical  report  prepared  for  the 
department  of  defense  focused  on  four  dimensions  of  SA  including  spatial,  identity, 
automation,  and  temporal  awareness  (Harwood  et  al.,  1988). 

b.  The  Need  for  SA  or  the  Lack  Thereof 

SA  is  essential  for  good  decision-making.  A  lack  of  it  can  have  disastrous 
consequences.  Dr.  Endsley  explains  that  “...problems  with  SA  were  found  to  be  the 
leading  causal  factors  in  a  review  of  military  aviation  mishaps,  and  in  a  study  of  accidents 
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among  major  air  carriers,  88%  of  those  involving  human  error  could  be  attributed  to 
problems  with  situation  awareness”  (Endsley,  1999  ,  p.  1).  Furthermore,  though  some  of 
the  aviation  mishaps  were  found  to  be  the  result  of  incorrect  action  selection,  they  are 
actually  errors  in  SA  (Endsley,  1999).  The  author  stated  that  the  aircrews  made  the 
correct  decision  based  on  their  mental  picture  of  the  situation,  but,  since  that  picture  was 
inaccurate,  they  made  erroneous  decisions  that  resulted  in  the  mishap. 

The  aviation  community  is  not  the  only  group  to  have  suffered  from  issues 
resulting  from  a  lack  of  SA.  There  have  been  studies  dealing  with  automobile  driver  SA 
as  well.  Ma  &  Kaber  (2005)  conducted  a  study  regarding  the  effects  of  an  adaptive 
cruise  control  (ACC)  and  use  of  a  cell  phone  while  driving.  Half  of  their  participants 
were  required  to  use  cell  phones  while  driving  virtual  cars  in  a  driving  simulation  with 
the  use  of  ACC.  The  other  half-used  cell  phones,  but  without  the  ACCs.  The  results 
indicated  that  use  of  the  ACC  system  improved  SA  during  the  driving  task  and  reduced 
driver  mental  workload.  The  cell  phone  conversation  caused  negative  effects  on  driver 
SA  and  increased  driver  mental  workload.  This  study  suggests  that  the  introduction  of 
automation  in  the  driving  task  frees  up  mental  processes  that  the  driver  can  bring  to  bear 
on  other  driving  requirements.  Therefore,  automobile  automation  can  significantly 
improve  driver  SA.  The  implication  of  this  study  is  that  a  lack  of  SA  while  operating  a 
motor  vehicle  can  increase  the  frequency  of  accidents. 

Not  only  is  SA  essential  in  aviation  tasks  and  while  operating  automobiles, 
it  is,  also,  an  important  part  of  ground  combat.  Mathews  et  al.  (2000)  states  that  SA  is  a 
fundamental  aspect  of  infantry  operations  and  discusses  a  systematic  approach  to 
determine  infantry  SA  requirements.  According  to  these  researchers,  the  emerging 
infantry  doctrine  and  non-linearization  of  the  modem  battlefield  requires  soldiers  to 
quickly  assess  and  act  based  on  their  available  information.  Finally,  this  study 
emphasizes  that  the  ground  military  troops  need  good  SA  if  they  are  to  be  successful  on 
today’s  complex  battlefield. 

c.  Endsley ’s  Model  And  The  Levels  of  SA 

Endsley  and  Garland  (2000)  defined  SA  in  a  three  level  model: 
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Level  1  -  Perception:  This  is  the  foundational  level  on  which  the  two 
subsequent  levels  rest.  The  perception  of  the  status,  attributes,  and  dynamics  of  the 
elements  in  the  environment  is  the  very  first  level  of  SA.  Without  basic  knowledge  of  the 
environment,  it  is  highly  possible  that  the  person  will  have  an  incorrect  picture  of  his 
environment,  which  could  lead  to  disastrous  consequences. 

Level  2  -  Comprehension:  SA  cannot  be  confined  only  to  one’s 
perception  of  the  world.  It  goes  beyond  simple  awareness  of  the  environment.  This 
requires  integration  of  different  information  from  multiple  sources.  This  level,  also, 
requires  mental  processing  of  the  information  by  choosing  the  right  components  based  on 
the  goal. 

Level  3  -  Projection:  This  is  the  third  and  the  highest  level  of  SA.  Level- 
3  SA  is  defined  by  Endsley  as  the  ability  to  project  forward  from  current  events  and  to 
anticipate  future  occurrences.  She  emphasizes  that  this  level  allows  the  person  to  make 
rapid  decisions  by  using  his  expectation  of  what  might  happen  in  the  future.  Figure  3  is 
Endsley’ s  Model  of  Situational  Awareness  in  Dynamic  Decision-Making  in  which  all 
levels  of  SA  are  components. 
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Figure  3.  Model  of  Situational  Awareness  (After:  Endsley  and  Garland,  2000) 

In  their  book,  Human  Factors  Engineering,  Wickens  et  al.  (2004),  state 
that  selective  attention  is  an  important  element  in  perception  and  that  projection  heavily 
depends  on  working  and  long-term  memory. 

d.  SA  and  Decision-Making 

Endsley  and  Garland  (2000)  believe  that  SA  is  only  a  component  of  good 
decision-making  as  seen  in  Figure  3.  They  emphasize  that  SA  is  a  precursor  to  decision¬ 
making  and  point  out  that  there  are  other  factors  that  influence  SA  and  that  can  contribute 
to  a  good  performance.  Endsley  (1995)  explains  that  making  an  incorrect  decision  is 
possible  even  though  there  is  perfect  SA.  Endsley  gives  an  example  of  a  military 
commander  who  has  a  perfect  SA  of  the  battlefield,  but  selects  an  inappropriate  course  of 
action  (as  cited  in  Endsley  and  Garland,  2000). 
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Dr.  St.  John  et  al.  (2000),  in  a  U.S.  Navy  SPAWAR  technical  report, 
explains  the  relationship  between  SA  and  tactical  decision-making  after  they  conducted 
two  experiments.  The  subjects  in  their  study  were  28  Marine  Corps  officers  and  senior 
enlisted  staff  non-commissioned  officers. 

In  their  first  experiment,  they  hypothesized  that  “a  good  graphical 
representation  leads  to  a  better  recall  of  enemy  intent  and  future  position  than  either 
textual  or  implicit  representation  of  that  information  on  a  battlefield  map”  (St.  John  et  al., 
2000).  The  researchers  provided  this  information  to  the  participants  under  two  methods. 
Method  1  was  text  descriptions  of  unit  locations,  size,  and  direction  of  movement. 
Method  2  showed  participants  representations  of  enemy  location,  size,  and  direction  of 
movement  on  a  high  quality  military  grid  map.  After  some  time,  the  subjects  were 
required  to  provide  this  information  back  to  the  researchers.  The  researchers  did  not  find 
any  statistical  significance  between  the  two  methods  of  depicting  the  basic  SA 
information. 

In  their  second  experiment,  they  tried  to  answer  the  question:  How  does 
uncertainty  affect  tactical  decision-making  and  how  can  its  adverse  effects  be  minimized 
or  eliminated?  Their  hypothesis  was  that  increasing  situation  uncertainty  would  lead  to  a 
more  wait  and  see  approach  and,  consequently,  to  a  slower  decision-making  process. 
Furthermore,  they  thought  that  the  effects  of  situation  uncertainty  would  lead  to  longer 
decision-making  times  in  inexperienced  subjects  versus  experienced  ones.  Contrary  to 
their  expectation,  rank  and  military  experience  did  not  predict  the  quality  of  their  decided 
courses  of  action.  The  amount  of  experience  that  subjects  had  within  a  Combat 
Operations  Center  (COC)  was  the  deciding  factor  with  regard  to  the  quality  of  their 
decided  course  of  action.  Their  observation  was  that  the  participants  who  had  less  COC 
experience  chose  the  wait-and-see  approach,  while  the  participants  with  more  COC 
experience  were  found  to  be  less  affected  by  the  uncertainty  in  the  scenarios. 

Stanners  and  French  (2005)  did  a  study  on  the  relationship  between  SA 

and  decision-making.  The  number  of  participants  was  24.  They  decided  to  use  a 

commercial  off-the-shelf  first  person  shooter  computer  game  called  Operation 

Flashpoint.  Participants  were  provided  with  a  summary  of  a  combat  situation.  The 
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researchers  assessed  SA  by  using  the  direct  questioning  technique  (DQT).  This  consisted 
of  asking  questions  to  the  participants  at  natural  breaks  during  the  game.  SMEs  were 
responsible  for  assessing  decision-making  among  the  participants.  The  SMEs  scored  the 
participants  as  optimal  (5),  average  (3),  or  poor  (1).  This  is  a  typical  1-5  Likert  scale. 
They  used  the  action-inference  decision  tree  (AIDT)  to  cover  every  possible  action  that  a 
participant  could  take.  The  SME  plotted  the  participant’s  decision  on  the  chart.  They 
found  a  positive  correlation  between  SA  and  decision-making,  as  well  as,  SA  and 
planning. 


e.  Measuring  Individual  SA 

The  scientific  community  has  different  opinions  regarding  the  exact 
method  that  should  be  used  to  measure  SA  and,  as  a  result,  they  have  developed  several 
different  ways  to  measure  it.  Salmon  et  al.  (2004)  classified  different  techniques  to 
measure  SA: 

•  On-line  freeze  techniques.  Direct  measurement  of  the  subject  SA. 
Situational  Awareness  Global  Assessment  Technique  (SAGAT)  and 
Situational  Awareness  Rating  Technique  (SART)  are  the  techniques  that 
are  widely  validated. 

•  Real-time  probe  techniques.  There  is  no  freeze  of  the  task.  This 
technique  involves  SA  related  queries  during  the  task. 

•  Self-Rating  techniques.  After  the  task,  the  subjects  rate  his/her  own  SA  on 
a  scale. 

•  Observer  rating  techniques.  There  are  subject  matter  experts  (SMEs)  who 
evaluate  the  SA. 

•  Questionnaire  techniques.  SA-related  questionnaires  are  provided  for  the 
subjects  after  the  task.  Then,  the  subjects  evaluate  themselves. 

SAGAT  is  the  technique  that  was  developed  by  Mica  Endsley.  Endsley  et  al.  (2000) 
explains  that  the  SAGAT  procedure  involves  stopping  exercises  and  simulations  in  a 
variety  of  ways;  then,  asking  the  participants  questions  that  are  designed  to  assess  the 
individual’s  SA.  The  questions  incorporate  perception,  comprehension,  and  projection. 
Another  popular  measurement  technique  is  SART.  SART  is  generally  used  in  the 
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aviation  society.  Endsley  (1998)  explains  that  SART  provides  subjective  ratings  of  SA 
by  the  operator.  SART  ratings  are  highly  correlated  with  operator  workload  and 
performance. 

The  various  SA  assessment  techniques  can  all  yield  valid  results.  The 
proper  tool/technique  is  a  function  of  the  type  of  task  and  the  domain  of  the  users 
involved.  Based  on  the  specific  situation,  the  researchers  choose  the  technique. 

f  Measuring  Team  SA 

Many  research  teams  and  organizations  have  conducted  experiments  on 
measuring  individual  SA  in  military  environments.  These  evolved  into  a  need  to  measure 
SA  collectively  within  a  team.  Salmon  et  al.  (2004)  did  a  study  focused  on  finding  the 
best  measurement  device  for  SA  in  a  Command,  Control,  Computers,  and 
Communication  (C4)  environment.  They  indicated  that  the  existing  techniques  used  to 
assess  individual  SA  were  adequate.  However,  there  was  a  need  for  some  mechanism  to 
assess  team  SA.  One  of  their  conclusions  was  that  a  combination  of  techniques,  such  as 
the  Situational  Awareness  Rating  Scales  (SARS)  and  SAGAT,  would  be  a  suitable  way 
of  assessing  SA  for  C4  teams.  Another  study,  entitled  Development  of  a  Metric  for 
Collaborative  Situation  Awareness  (CSA),  focused  on  military  SA.  In  this  study,  Redden 
et  al.  (2005)  used  PC-based  game  scenarios,  which  focused  on  actual  military  operations 
in  urban  terrain  (MOUT)  as  their  platform  for  the  study.  Their  goals  were  to: 

•  Design  MOUT  scenarios  to  elicit  realistic  communications  and  tactics, 

•  Develop  a  script-based  assessment  of  the  team  member’s  and  the  team 
leader’s  SA  at  the  three  levels  of  SA, 

•  Manipulate  communication  structures, 

•  Assess  the  degree  to  which  individual  SA  and  the  communication 
structures  affect  the  SA  of  the  leader. 

The  three-person  teams  used  in  the  study  played  the  roles  of  infantry 
squad  leader  and  two  subordinate  team  leaders.  The  researchers  measured  SA  levels  for 
both  the  team  and  squad  levels.  The  questions  that  the  researchers  used  to  evaluate  SA 
were  designed  to  distinguish  between  the  three  levels  of  SA  (Perception,  Comprehension, 
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and  Projection).  The  questions  focused  on  the  squad  and  fire-team  levels  of  leadership. 
This  study  is  a  good  example  of  how  to  look  at  team  CSA  using  a  simulation  or  a  game- 
based  environment. 

We  can  see  that  situational  awareness  is  a  vital  component  of  decision¬ 
making  within  military  aviation,  as  well  as,  for  military  ground  training  and  combat.  The 
military  has  several  decision-making  tools  that  might,  also,  be  used  for  evaluating  SA. 

4.  Tactical  Decision-Making  Game  (TDG) 

The  tactical  decision  games  (TDGs)  are  situational-based  scenarios  where 
individuals  are  required  to  exercise  mental  agility  to  meet  the  demands  of 
the  situational  stimuli  while  implementing  a  problem  solving  solution. 

The  TDG  can  range  from  paper  media  to  a  situation  given  orally  to 
Marines  by  a  seminar  leader  or  facilitator.  (Marine  Corps  Institute). 

Military  organizations  have  been  using  TDGs  for  hundreds,  even  thousands  of 
years.  One  of  the  earliest  references  to  a  TDG  was  from  the  famed  Chinese  general  and 
theorist,  Sun  Tzu.  “This  technique  (TDG)  can  be  traced  back  at  least  to. ..Sun  Tzu,  who 
was  advocating  their  use  more  than  2,500,  years  ago.”  (U.S.  Marine  Corps,  1989).  More 
recently,  the  Prussian  state  in  the  1800s  was  noted  to  be  using  TDG.  Many  Prussian 
military  leaders  at  the  start  of  the  19th  century  believed  that  war  was  coming  soon.  Their 
military  was  faced  with  the  monumental  task  of  the  state’s  security  and  was  required  to 
quickly  develop  its  officers  and  enlisted  personnel.  J.T.  Gatto,  in  his  1991  book  entitled 
The  Prussian  Connection,  stated  that  the  Prussian  Army  had  to  “prepare  hard  in 
peacetime  to  be  ready  when  war  began.  From  the  very  beginning  of  a  Prussian  (later 
German)  cadet’s  career,  TDGs  were  used  to  sharpen  the  students’  decision-making  skills 
and  to  provide  a  basis  for  evaluating  them  on  their  character”  (as  cited  in  Vandegrift, 

2006,  p.  32). 

Modem  TDGs  are  usually  conducted  with  paper  and  pencil  around  a  sand  table  or 
some  other  terrain  model.  They  can,  also,  be  conducted  without  a  terrain  representation. 
To  improve  situational  awareness,  the  modem  military  uses  TDGs  primarily  to  build 
tactical  decision-making  skills  and  to  assist  military  personnel  with  critical  thinking 
(Gonsalves,  1997). 
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Some  benefits  of  TDGs  include: 

•  Improving  pattern  recognition  skills, 

•  Exercising  the  decision-making  process, 

•  Improving  and  practicing  communication  skills, 

•  Increasing  leadership  potential. 

Some  limitations  of  the  TDG  are: 

•  Representing  a  snapshot  in  time, 

•  Allowing  participants  to  make  only  one  move, 

•  Simulating  operating  environment  is  difficult  -  impossible  to  simulate  the 
friction  and  uncertainty  of  the  operating  environment, 

•  Working  for  units  no  larger  than  company  level, 

•  Applying  to  special  operations  is  difficult  (Marine  Corps  Institute,  1989). 

5.  Tactical  Decision-Making  Simulation  (TDS) 

As  computer  technology  became  more  prevalent  and  more  powerful  throughout 
the  1990s,  TDGs  began  to  give  way  to  computer-based  tactical  decision-making 
simulations  (TDSs).  TDSs  kept  the  basic  premise  of  the  TDG,  but  integrated  computer 
technology  with  artificial  intelligent  agents.  This  new  approach  allowed  the  Marine,  or 
team  of  Marines,  to  fight  against  the  system  that  had  an  artificial  intelligence  component. 
Historically,  the  military  has  been  one  of  the  early  adopters  in  the  use  of  computer-based 
simulations  applied  in  a  training  environment.  However,  there  is  still  much  to  learn  about 
how  to  best  use  them  in  a  military  training  curriculum. 

Over  the  last  two  decades,  all  branches  of  the  military  service  have  begun  to 
express  more  interest  in  using  simulation  to  support  the  training  of  ground  officers. 
Desktop  simulations  and  digital  game-based  technologies  have  earned  much  attention  for 
their  potential  as  training  interventions.  Supporters  view  the  sensory  (predominantly 
visual  and  auditory)  interactivity  of  the  technologies  as  powerful  means  of  fostering  the 
development  of  cognitive  skills  relative  to  the  task. 

Ground  officers  work  in  environments  that  are  very  fluid  and  dangerous;  their 
tasks  are  less  procedural  and  more  subject  to  changing  situations.  TDSs  allow  them  to 
experience  near  real-world  tactical  situations  that  offer  the  opportunity  to  make  numerous 
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decisions  without  the  risk  of  harm.  Additionally,  unlike  a  real-world  training  scenario, 
these  decisions,  and  their  consequences,  can  be  replayed  many  times  to  the  decision¬ 
maker  as  a  training  tool.  This  can  illustrate  what  went  right  and  what  may  have  gone 
wrong. 

The  research  on  TDSs  has  mostly  dealt  with  overall  teaching  methodologies 
involving  simulation,  computer-based  training,  and  commercial  games  used  for  training. 
The  preponderance  of  the  scientific  literature  on  this  subject  has  been  devoted  to  macro- 
cognitive  activities,  decision-making,  situational  awareness,  and  training  effectiveness. 
Researchers  have  focused  on  dealing  mostly  with  overall  learning  and  retention  provided 
by  the  simulation  (Baxter,  Ross,  Phillips,  Shafer,  and  Fowlkes,  2004)  and  (Fowlkes, 
Cohn,  Jones,  Hafich,  Nichols,  Woodman,  and  Bushika,  2005).  One  area,  in  which  many 
of  these  studies  are  lacking,  is  an  emphasis  on  learning  to  use  (operate)  particular 
simulations. 

Some  simulations  require  many  hours  to  bring  students  to  a  level  where  they  can 
operate  them.  This  phase  is  the  “ learn  the  system ”  phase,  which  is  an  essential  piece  of 
the  overall  training  strategy.  Since  military  training  time  is  always  limited,  it  is  vital  for 
students  to  learn  how  to  use  the  simulation  in  the  shortest  period.  This  is  so  that  they  can 
begin  progressing  quickly  toward  the  mastery  of  their  learning  objectives,  instead  of 
hunting  and  pecking  their  way  through  the  session.  This  wastes  time  and  resources.  A 
lack  of  meticulous  work  in  this  area  (examining  preliminary  training  needs)  could  cause 
the  introduction  of  TDS  technology  that  does  not  capitalize  on  its  training  potential. 

In  their  Naval  Postgraduate  School  thesis  on  leveraging  commercial  off  the  shelf 
games  for  use  in  military  environments,  Nolan  and  Jones  (2005)  conducted  several 
training  sessions.  Initially  they  used  the  lecture  method  and,  to  a  lesser  degree,  hands  on 
computer  control  of  their  avatar  (the  virtual  representation  of  a  human  within  the  game): 

We  initially  felt  that  the  training  we  provided  and  their  (the  student’s) 
exposure  to  the  game  environment  would  be  sufficient.  Survey 
results... indicated  that  the  participants  felt  that  there  was  too  much  lecture 
and  not  enough  hands-on  training. .  .Based  on  this  feedback,  we  altered  our 
training  for  the  remaining  two  pilot  studies.  Our  familiarization  and 
training  program  evolved  to  a  much  shorter  classroom  presentation 


32 


followed  by  a  group,  hands-on  training  session  in  the  virtual 
environment.... The  positive  results  from  these  pilot  studies  led  us  to  use 
this  ads  our  training  and  familiarization  technique  during  the  experiment 
conducted  at  Ft  Benning,  GA  (Nolan  &  Jones,  2005). 

Based  on  the  previous  research,  we  can  see  that  a  variety  of  methods  have  been 
used  to  teach  participants  to  use  a  simulation.  However,  the  most  effective  one 
mentioned  by  these  researchers  is  one  where  there  is  a  short  lecture  phase  preceding  a 
“hands-on”  training  phase.  This  is  where  participants  actually  operate  the  simulation  on 
their  own. 

6.  Comparing  TDGs  and  TDSs 

The  TDG  and  the  TDS  have  many  similarities,  but  there  are  a  number  of 
differences  as  well.  Baxter  et  al.  (2004)  conducted  a  study  that  assessed  the  CCM  TDS 
against  the  paper  based  tactical  decision  game  (TDG).  The  researchers  had  fourteen 
participants  from  the  Infantry  Platoon  Sergeant  Course  taught  at  the  Advanced  Infantry 
Training  School  at  Camp  Geiger,  North  Carolina.  They  used  a  within-subjects  design, 
where  all  participants  went  through  both  the  TDS  and  TDG.  The  researchers  gave 
surveys  to  the  students  and  instructors  before  and  after  the  TDS  and  the  TDG  sessions. 
In  the  end,  the  researchers  concluded  that  TDS  was  most  helpful  for  developing  the 
students’  mental  coordination,  mental  simulation,  and  uncertainty  management  tasks; 
TDS  was  observed  to  be  more  helpful  for  planning.  The  researchers,  also,  examined 
decision-making  and  its  relationship  to  the  TDG/TDS.  However,  that  proved 
inconclusive  because  neither  TDS  nor  TDG  was  a  recommended  teaching  method.  More 
research  is  definitely  needed  to  explore  the  ability  of  TDG  and  TDS  to  train  and  evaluate 
the  skills  that  they  are  believed  to  train. 

7.  Close  Combat  Marines  (CCM) 

The  Marine  Corps  has  been  leveraging  aspects  of  digital-game  based  methods  and 

inserting  them  into  the  tactical  decision-making  simulations  (TDSs)  to  supplement 

existing  training  (Baxter,  Ross,  Phillips,  Shafer,  and  Fowlkes,  2004,  p.  1).  The  Marine 

Corps  currently  uses  a  TDS  named  Close  Combat  Marine  (CCM)  at  various  leadership 

and  tactical  training  schools.  This  TDS  is  a  top-down  view  of  the  battlefield  and  was 
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designed  to  develop  the  tactical  decision-making  skills  of  entry-level  officers.  The  CCM 
used  at  the  introductory  infantry  training  school  in  Quantico,  Virginia,  is  called  The  Basic 
School  (TBS).  TBS’s  follow-on  school  for  advanced  infantry  tactics  is  the  Infantry 
Officer’s  Basic  Course  (IOBC),  also  located  in  Quantico,  Virginia,  uses  this  product  as 
well.  CCM  is  not  exclusively  used  for  training  U.S.  Marine  Corps  officers.  Enlisted 
Marines,  also,  gain  exposure  to  this  tool  at  the  Platoon  Leader’s  Course  (PLC)  for  non¬ 
commissioned  officers  (NCOs)  at  Camp  Geiger,  North  Carolina  (Nichols,  2006). 

CCM  has  a  series  of  setup  and  information  screens  as  seen  in  Figure  4 


Figure  4.  CCM  Startup,  Mission,  and  Results  screens 

The  final  screen  displayed  captures  the  results  from  the  scenario  and  displays 
these  to  the  TDS  participants.  Each  CCM  scenario  comes  with  a  recommended  number 
of  troop  and  vehicle  assets.  Each  scenario  is  provided  with  two  operations’  orders,  one 
for  the  offense,  and  one  for  the  defense.  Finally,  CCM  provides  a  specific  terrain  that  is 
unique  to  the  scenario  in  which  the  teams  engage  one  another  and  attempt  to  accomplish 
their  missions. 
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III.  EXPERIMENTS 


A.  EXPERIMENT  1  -  PRE-TRAINING  STUDY  LEARN  THE  SYSTEM 

1.  Introduction 

Based  on  the  findings  from  our  literature  review,  we  believe  it  was  important  to 
establish  a  base  of  knowledge  among  our  participants.  It  is,  also,  important  to  provide 
them  with  an  opportunity  to  learn  how  to  operate  the  TDS  system  that  will  be  used  in  our 
second  study.  To  accomplish  this,  we  designed  a  study  to  explore  preliminary  training 
approaches  to  see  which  method  brought  the  users  to  an  acceptable  level  of  competency. 

In  a  U.S.  Army  Command  and  General  Staff  College  Master’s  Thesis  entitled, 
The  Application  of  Off-the-Shelf  Military  Simulations  to  Train  Decision-Making  and 
Teach  Tactics ,  Shoemaker  (2003)  analyzed  several  different  computer  simulations.  His 
study  required  officers  to  evaluate  these  simulations  on  their  own  time  and  report  their 
findings  back  to  him  based  on  a  template  that  was  provided.  While  the  specific  results 
are  interesting,  they  are  not  relevant  to  our  work.  What  is  interesting  is  that  no 
preliminary  training  was  provided  at  all.  The  officers  were  simply  given  the  scenario  and 
told  to  figure  it  out.  Shoemaker  reports  that  there  was  a  variance  of  over  four  hours  in  the 
lengths  of  time  that  it  took  for  each  officer  to  evaluate  the  simulations  (Shoemaker, 
2003).  This  report  asserts  that,  given  no  initial  training  on  how  to  best  interact  with  the 
simulation,  wide  variations  are  possible  in  the  lengths  of  time  for  a  student  to  become 
familiar  with  a  system  or  tool.  It  is  likely  that  the  time  to  learn  to  operate  a  system 
effectively  is  highly  variable.  It  is,  also,  likely  that  this  time  period  will  considerably 
depend  on  the  type  of  system  that  needs  to  be  used  in  the  study. 

In  our  initial  study,  we  analyzed  and  identified  a  “best  practices”  approach  to 
introducing  to  users  a  simulation  called  Close  Combat  Marines  (CCM).  Though  CCM 
has  been  used  as  a  training  tool  for  approximately  ten  years  in  the  U.S.  Marine  Corps, 
there  is  still  a  question  about  the  best  way  of  introducing  it  to  new  users.  The  intent  of 
this  experiment  was  to  explore  two  issues:  Firstly,  the  amount  of  time  needed  to  learn  to 
operate  the  simulation  and,  secondly,  the  best  method  for  bringing  a  participant  with  no 
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experience  with  tactical  decision-making  simulations  to  a  place  where  he  would  be 
competent  and  able  to  participate  in  initial  TDS  training.  Additionally,  we  used  this 
experiment  to  evaluate  their  experimental  methodology  to  gain  experience  going  through 
the  institutional  review  board  (IRB)  process,  experience  system  hardware  and  software 
setup,  and  practice  the  management  of  all  participants  throughout  the  entire  process. 

2.  Procedure 

We  provided  a  very  short  (no  more  than  five  minutes)  verbal  explanation  of  the 
overall  experimental  scenario.  The  three  methods  in  which  the  students  participated 
were: 

•  “Boot  Camp  Only”  -  The  manufacturer  provided  a  Boot  Camp  scenario 
designed  specifically  to  train  a  user  to  operate  the  simulation; 

•  “Memory  Sheet  Only”  -  A  memory  sheet  that  outlines  all  functionality  of 
the  simulation  controls  and  describes  the  various  parts  of  the  screen  in 
terms  of  menus  and  displays; 

•  “Boot  Camp  Plus  Memory  Sheet”  -  A  combined  approach  of  the  Boot 
Camp  and  the  Memory  Sheet.  Our  null  hypothesis  was  that  there  was  no 
difference  between  any  of  the  three  training  methodologies. 

Participants  engaged  in  and  were  evaluated  on  only  one  of  the  three  above 
scenarios  necessitating  a  between-subjects  design.  The  following  research  questions 
were  central  to  this  investigation: 

•  What  is  the  best  method  to  pre-train  users  to  operate  CCM  effectively  on 
their  own? 

•  Should  this  simulation  always  be  used  in  a  controlled  instructor-led 
environment  or  should  the  students  be  allowed  to  explore  and  learn  on 
their  own? 

•  How  much  simulation  time  is  needed  to  accomplish  this  prior  to  the  start 
of  the  actual  training,  i.e.,  how  long  should  the  introductory  training 
sessions  last? 

•  How  many  training  sessions  are  necessary  before  the  user  is  ready  for  a 
real  TDS  training  scenario? 

•  Is  a  single  training  methodology  adequate  or  should  a  combination  of 
methodologies  be  used? 

Through  this  effort,  we  sought  to  obtain  an  improved  set  of  introductory  training 

approaches  that  will  be  incorporated  into  a  future  follow-on  study. 
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Ho  (Null  Hypothesis)  =  There  is  no  difference  between  the  abilities  of  the  three 
methods. 

Ha  (Alternative  Hypothesis)  =  There  is  some  difference  between  the  three 
methods  with  respect  to  the  number  of  overall  mistakes  that  each  participant  will  make. 
We  chose  an  a  =  0.05  as  the  significance  criteria  whether  to  reject  the  null  hypothesis. 

3.  Method 

a.  Participants 

The  participants  were  all  students  of  the  Naval  Postgraduate  School 
(N=12),  but  ranged  from  among  three  different  academic  curricula.  There  were  nine 
participants  studying  Human  System  Integration  (HSI);  one  in  Systems  Engineering  (SE); 
and  two  from  the  Modeling  Virtual  Environments  and  Simulation  (MOVES)  program. 
The  HSI  program  is  considered  non-technical  and  the  SE  and  MOVES  degrees  are  both 
technical.  All  twelve  participants  were  U.S.  Naval  officers  (9  males  and  3  females). 
Participants  were  randomly  assigned  to  the  three  groups.  A  3-block  design  was  used  to 
ensure  even  distribution  to  assigned  groups.  We,  also,  collected  other  demographic 
information,  including  age,  dominant  hand,  and  total  time  in  service.  Participants  were 
all  treated  in  accordance  with  ethical  standards  established  by  the  American 
Psychological  Association.  Further,  our  study  design  was  approved  by  the  IRB 
committee  at  NPS.  The  IRB  paperwork  is  included  in  Appendix  B. 

b.  Apparatus 

This  experiment  was  conducted  in  the  Human  Systems  Integration  Lab 
(HISL)  at  the  Naval  Postgraduate  School,  Monterey,  California.  The  layout  of  the  space 
used  during  this  study  can  be  seen  in  Figure  5. 
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Figure  5.  Experiment  1  -  Lab  Setup 

We  used  four  computer  desktop  workstations  consisting  of  a  CPU 
(Pentium  4),  19  inch  monitor,  a  standard  U.S.  English  keyboard,  and  a  mouse.  All  of 
these  workstations  were  wired  into  the  schools  network  and  had  CCM  installed  on  them. 
We  used  a  Canon  Digital  Video  Camcorder,  with  wide  screen  recording  and  lOx  optical 
zoom,  to  record  the  screen  that  the  participants  were  using  throughout  their  test  scenario. 
This  camcorder  captured  audio  of  the  participants  for  use  in  the  verbal  protocol  analysis. 
Due  to  scheduling  conflicts  with  other  classes,  we  were  only  able  to  run  one  participant  at 
a  time  through  the  experiment.  For  a  single  participant,  we  used  two  computers.  On  one 
computer,  we  had  a  standardized  PowerPoint  presentation  that  provided  instructions 
specific  to  each  of  the  three  different  training  methodologies.  On  the  other,  they  used 
CCM  for  the  training  and  testing  phases  of  the  experiment. 

c.  Procedure 

This  study  was  a  between-subjects  1x3  design.  The  independent  variable 
was  the  method  of  training  and  the  dependent  variable  was  the  number  of  mistakes 
participants  made  (based  on  analysis  of  their  ‘think  aloud’  protocol  and  the  analysis  of 
the  audio/video  recordings  of  their  test  sessions).  We  conducted  the  audio/video  analysis 
jointly. 

The  experiment  lasted  approximately  50-60  minutes  for  each  participant. 

Upon  arrival  at  the  lab,  participants  were  administered  a  survey  that  took  demographic 
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information.  Additionally,  we  asked  the  participants  about  prior  experience  with  CCM  or 
other  military  tactical  decision-making  simulations.  The  survey  required  students  to 
disclose  their  level  of  experience  with  commercial  computer  games  and  a  number  of 
other  variables  relative  to  their  professional  expertise.  All  participants  received  the  same 
questionnaire,  basic  introduction  to  the  experiment,  and  an  overview  of  CCM.  After  this 
was  completed,  the  participants  were  assigned  to  one  of  the  three  groups  (we  did  not 
disclose  the  purpose  of  the  experiment). 

(1)  “Boot  Camp  Only.”  The  first  group’s  training  scenario  task 
was  to  learn  to  use  the  simulation  only  by  exploring  and  going  through  the  developer’s 
installed  Boot  Camp  training.  This  segment  of  the  CCM  system  was  specifically 
designed  by  the  simulation  developers  to  provide  new  users  with  an  easy  way  to  learn 
how  to  move,  shoot,  issue  orders,  gather  information  regarding  the  health  and  welfare  of 
their  teams,  as  well  as,  to  generally  navigate  within  the  virtual  environment.  No  training 
aids  were  provided  to  the  Boot  Camp  group. 

(2)  “Memory  Sheet  Only.”  Group  2’s  training  task  was  to 
learn  to  use  the  simulation  by  entering  into  and  playing  one  of  the  scenarios  provided 
with  CCM.  The  users  used  CCM  and  were  expected  to  learn  its  functionalities  by  using 
the  system.  Each  person  in  this  group  was  provided  with  a  memory  sheet  that  included 
all  commands  that  they  would  need  to  operate  CCM  during  the  test  phase  (this  sheet  was 
created  by  summarizing  the  material  from  the  CCM  User  Manual).  The  article,  “Utility 
of  Game  Instruction”  (Chen,  2003),  inspired  the  use  of  a  memory  sheet. 

(3)  “Boot  Camp  &  Memory  Sheet.”  The  training  method  used 
with  Group  3  was  to  use  a  combination  of  the  Boot  Camp  and  the  Memory  Sheet. 

The  time  that  subjects  spent  in  each  condition  varied,  to  some  degree, 
between  participants  because  participants  in  training  methods  “Boot  Camp  Only”  and 
“Boot  Camp  &  Memory  Sheet”  were  somewhat  controlled  by  the  sequential  nature  of  the 
Boot  Camp.  The  Memory  Sheet  Only  method  was  much  more  free-form.  Participants 
spent  less  time  in  this  section  of  the  experiment  as  they  seemed  eager  to  get  to  the  test 
phase.  All  training  sessions  lasted  no  more  than  20  minutes.  If  students  expressed 


39 


confidence  in  their  ability  to  operate  the  simulation  prior  to  their  20-minute  training 
session  being  completed,  they  were  allowed  to  begin  the  test  scenario  at  that  time.  All 
test  scenarios  lasted  no  more  than  25  minutes.  We  stopped  the  scenario  if  the  participant 
had  not  received  a  Mission  Accomplished,  or  a  Mission  Failure  notification  by  that  time. 
Each  participant  went  through  the  same  test  scenario  “Module  3  Fight  3-la  Machine  Gun 
Defense.”  Due  to  their  actual  performance  during  the  scenario,  the  amount  of  time  could 
be  less  than  40  minutes  if  the  participant’s  military  forces  called  for  or  were  offered  a 
truce,  were  judged  as  absolutely  victorious,  or  were  annihilated  by  the  artificial 
intelligence  present  with  in  the  CCM  system. 

We  digitally  recorded  all  sessions  and  required  that  participants  verbalize 
their  thoughts  as  they  made  their  way  through  the  instructions.  They  were  asked  to  do 
the  same  during  the  actual  use  of  the  simulation  during  the  test  phase.  This  procedure 
was  necessary  to  capture  each  participant’s  intentions  and  to  compare  what  the 
participants  intended  to  do  with  what  they  actually  accomplished.  We,  also,  compared 
these  results  with  information  obtained  in  the  Pre-TDS  survey.  Upon  completion  of  their 
test  session,  participants  were  administered  a  Post-TDS  survey  in  which  they  entered  self- 
reported  values  to  determine  whether  they  felt  they  had  learned  specific  CCM  controls 
throughout  the  experiment. 

After  the  participants  had  departed,  we  reviewed  over  four  hours  of 
video/audio  recordings  and  analyzed  those  capturing  four  different  types  of  participant 
errors.  We,  also,  conducted  an  overall  assessment  of  their  simulation  strategies  with 
respect  to  how  they  used  the  simulation.  This  was  not  an  assessment  of  their  tactical 
infantry  decision-making  prowess.  Rather,  it  dealt  strictly  with  the  nature  of  the  ability  to 
control  different  aspects  of  the  simulation  and  to  get  the  information  that  they  needed  to 
make  decisions.  The  most  important  factor  that  we  analyzed  was  the  participant’s  ability 
to  accomplish  what  he  intended  to  after  completing  his  particular  method  of  training. 

4.  Results 

Our  primary  focus  was  to  determine  which  of  those  three  training  methods  proved 
to  yield  the  least  errors  during  the  test  scenario.  In  Table  2,  we  summarize  the  results  of  a 
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descriptive  analysis  of  the  methods.  The  “Memory  Sheet  Only”  group  yielded  the  fewest 
total  errors  on  average  between  the  groups  at  9.75.  They,  also,  had  the  fewest  errors 
when  considering  Error  Factor  at  2.6. 


Table  2.  Summary  Statistics 


Total  Errors  (Mean) 

14.25 

9.75 

18 

Total  Errors  (Std  Dev) 

10.59 

6.39 

4.83 

Error  Factor  (Mean) 

6.18 

2.6 

9.85 

Error  Factor  (Std  Dev) 

6.05 

1.97 

6.97 

Error  Factor  =  Total  Errors/Instructor  Evaluation 


Total  Errors  is  a  raw  score  that  resulted  from  the  simple  tabulation  of  errors  that 
the  participants  made  during  their  testing  phase.  This,  however,  proved  to  be  somewhat 
misleading  because  some  participants  were  much  more  active  in  giving  orders  to  their 
troops,  telling  them  to  move,  shoot,  ambush,  and  defend  many  more  times  than  other 
participants.  These  active  participants  not  only  gave  many  more  orders,  but,  in  doing  so, 
increased  their  chance  of  making  errors.  Consequently,  we  decided  to  assign  a  score  that 
would  take  the  activity  level  of  the  participants  into  account.  We  then  divided  the  Total 
Number  of  Errors  by  this  score,  which  resulted  in  the  Error  Factor. 

Inferential  statistics  yielded  no  significant  difference  between  the  three  training 
methods.  We  used  an  F-Test  (ANOVA)  to  test  our  hypothesis.  This  resulted  in  p  = 
0.3563  for  total  errors  and  p  =  0.2241  for  Error  Factor.  Neither  of  these  is  sufficient  to 
reject  our  null  hypothesis  (Ho  =  No  difference  between  any  of  the  methods). 

In  addition  to  the  primary  hypothesis  tests  for  the  overall  study,  we  did  find  some 
other  interesting  results.  The  participants  who  had  a  high  level  of  previous  commercial 
video  game  experience  made  significantly  fewer  errors.  The  sex  of  the  participants,  also, 
proved  statistically  significant:  Males  performed  better  than  females.  Finally,  there  was 
a  significant  result  between  gaming  experience  and  sex  indicating  that  males  spent  more 


41 


time  playing  video  games  than  females.  However,  this  study  only  had  twleve  participants 
(3  were  female)  and  this  low  number  could  have  significantly  affected  the  results. 
(Detailed  statistical  output  is  provided  in  Appendix  E  at  the  end  of  this  document). 

5.  Summary 

The  analysis  of  the  data  collected  during  the  study  led  us  to  conclude  that  we  were 
unable  to  reject  our  null  hypothesis.  However,  there  were  other  interesting  findings  that 
arose  from  a  more  detailed  analysis  of  our  data  with  respect  to  how  other  dependent 
variables  influenced  total  errors  and  total  errors  with  factor.  One  interesting  finding,  that 
caused  us  to  come  up  with  the  factor  in  the  first  place,  was  that  female  participants  were 
making  very  few  errors.  On  the  surface,  it  appeared  that  they  were  performing  almost 
flawlessly.  Upon  scrutiny,  however,  the  reason  that  so  few  errors  were  made  was 
because  very  few  commands  were  issued.  They  issued  far  fewer  commands  than  males 
and,  therefore,  made  fewer  errors. 

There  was  one  critical  piece  of  data  that  we  believe  could  have  had  significant 
influence  on  the  total  number  of  errors.  However,  the  survey  that  solicited  this 
information  was  ambiguous  and  confusing  and  was,  therefore,  removed  from 
consideration.  This  variable  was  total  amount  of  video  game  experience.  It  was  not 
asked  in  terms  of  a  single  metric,  such  as  number  of  hours/week.  The  question  was 
structured  in  such  a  manner  that  the  participants  would  likely  not  know  whether  they 
were  answering  in  hours/day,  hours/week,  hours/month,  or  hours/year.  This  produced 
wild  variations  in  the  data  from  this  question  and,  even  with  extensive  effort  to  salvage 
the  data;  we  decided  that  it  would  be  too  misleading  to  report  it. 

It  is  interesting  to  note  that  the  Memory  Sheet  method  yielded  the  fewest  errors  in 
both  the  Total  Error  score,  as  well  as,  the  Error  Factor.  The  Memory  Sheet  method  was 
the  method  that  allowed  the  participants  to  engage  in  a  real  scenario  versus  going  through 
the  simulation  developer-provided  boot  camp.  We  noticed  that  when  the  memory  sheet 
was  provided  in  methods  2  and  3,  it  was  mostly  unused  by  participants.  After  the 
scenario,  many  of  the  participants  mentioned  they  totally  forgot  the  memory  sheet  as 
soon  as  they  started  taking  fire  from  the  enemy.  We  believe  that  the  memory  sheet  was 


42 


mostly  irrelevant  to  any  learning  that  took  place.  Further,  we  believe  that  the  Memory 
Sheet  Only  method  was  the  best  method  for  training  because  it  most  closely  aligned  with 
how  users  would  use  the  simulation  during  the  test  scenario.  They  were  required  to  issue 
orders,  view  toolbars,  and  assess  their  teams  in  exactly  the  same  manner  that  they  would 
be  required  to  during  the  test  phase.  In  effect,  they  were  practicing  how  they  would  use 
the  simulation.  This  important  training  concept  was  not  present  in  either  training  method 
one  or  three  where  the  boot  camp  was  the  primary  instructional  tool. 

Based  on  the  information  obtained  in  this  study,  we  can  offer  preliminary  answers 
to  our  research  questions: 

(1)  What  is  the  best  method  to  pre -train  users  of  the  TDS  to  operate  it 
effectively  on  their  own?  The  answer  to  this  question  was  inconclusive  based  on  our 
findings. 

(2)  Should  this  simulation  always  be  used  in  a  controlled  instructor-led 
environment  or  should  the  students  be  allowed  to  explore  and  learn  on  their  own?  We  do 
not  believe  that  the  former  should  always  be  the  case.  We  found  that  instructor-led 
training  would  be  beneficial  only  as  an  introduction  to  the  TDS  itself.  This  should  be 
very  brief.  After  that,  the  student  should  be  placed  in  front  of  a  computer  with  the  TDS 
installed  and  allowed  to  go  through  a  simple  module,  such  as  Module  1  Fight  1-1  a, 
Machine  Gun  Defense.  The  instructor  would  be  either  beside  or  behind  the  student  to 
provide  immediate  answers  to  the  student’s  questions  on  the  conduct  of  the  TDS.  This 
approach  allows  the  student  to  explore  the  simulation  scenario  and  forces  him  to  use  as 
many  resources  as  possible  to  accomplish  his  mission. 

(3)  How  much  simulation  time  is  needed  to  accomplish  this  prior  to  the  start 
of  the  actual  training,  i.e.,  how  long  should  the  introductory  training  sessions  last?  The 
questionnaire  results  varied  widely  from  “weeks”  to  “another  10-20  minutes.”  The  latter 
is  probably  closer  to  the  actual  answer.  We  believe  that  the  absolute  lack  of  prior  gaming 
experience  from  the  participant,  who  stated  “weeks”  would  be  required  to  gain 
proficiency,  was  exaggerating  due  to  obvious  frustration  with  his  lack  of  understanding 
and  ability  to  accomplish  his  intentions  throughout  the  test  phase. 
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(4)  Is  a  single  training  methodology  adequate  or  should  a  combination  of 
methodologies  be  used?  We  believe  that  a  combination  of  methodologies  should  be  used. 
Users  should  be  introduced  with  a  brief  overview  of  CCM  via  a  lecture  method 
accompanied  by  several  carefully  crafted  PowerPoint  slides  depicting  images  of  the  TDS 
in  various  stages.  The  various  screens  could  be  introduced  along  with  the  overall  goal  of 
the  simulation.  This  should  last  no  more  than  5-10  minutes.  The  learning  process  should 
consist  of  an  instructor  taking  questions,  but  getting  the  users  behind  a  station  as  quickly 
as  possible.  It  is  important  to  introduce  them  to  a  simple  scenario  and  let  them  explore. 
This  provides  them  with  expert  guidance  and  answers  to  their  questions  as  they  arise. 

(5)  How  many  training  sessions  are  necessary?  On  average,  participants 
stated  that  only  1-2  sessions  would  be  required  to  master  the  basics  of  operating  the 
CCM  controls.  We  agree  with  this  estimate  because  the  participants’  opinions  were 
confirmed  by  our  analysis  of  the  data. 

One  consideration  for  future  work  might  be  to  examine  some  elements  of 
personality,  such  as  extroversion  and  openness  to  see  if  they  are  influential  on  learning  to 
use  the  controls  within  a  TDS.  We  were  struck  by  how  some  participants  were  very 
timid  with  the  interface  and  consistently  stated  how  unsure  they  were  of  what  to  do  and 
how  to  do  it.  Others  took  control  immediately  and  began  to  issue  orders  to  CCM  teams 
in  a  chaotic  and  haphazard  way,  even  telling  friendly  teams  to  fire  on  other  friendlies. 
There  was  no  comprehension  about  who  was  who  or  what  the  overall  goal  of  the  scenario 
was  (even  though  it  had  been  explained  in  great  detail  during  the  scenario  setup  phase). 

The  results  from  the  experiment  were  very  influential  when  we  designed 
experiment  2.  We  drew  on  lessons  learned  to  develop  a  brief  instructor-led  introduction 
to  the  TDG/TDS  and  what  could  be  expected  in  each  session.  We,  then,  allowed  the 
students  to  engage  in  their  particular  roles  as  quickly  as  possible.  We  remained  close  by 
to  answer  questions.  This  was  the  intent  of  session  1  in  both  the  TDG  and  the  TDS. 
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B.  EXPERIMENT  2  -  PILOT  STUDY  EVALUATING  LEADERSHIP, 

DECISION-MAKING,  AND  SITUATIONAL  AWARENESS  WITH  TDG 
AND  TDS 

This  section  describes  our  second  experiment  where  our  intent  was  to  examine 
leadership,  decision-making,  and  situational  awareness  in  virtual  environments  and  how 
this  differed  from  the  traditional  TDG  training  method. 

1.  Introduction 

Tactical  decision-making  games  and  simulations  have  been  used  for  some  time  in 
the  training  of  ground  officers,  but  to  what  extent  they  could  be  used  to  evaluate 
leadership,  decision-making,  and  situational  awareness,  is  largely  unknown.  Some 
studies  have  explored  decision-making  and  situational  awareness  with  respect  to  TDSs, 
but  even  fewer  examined  the  leadership  question.  The  TDG  has  been  used  by 
professional  military  schools  training  junior  commissioned  officers  and  non¬ 
commissioned  officers.  This  training  method  has  been  proven  effective  in  evaluating 
decision-making  based  on  the  evaluations  of  subject  matter  experts.  The  Tactical 
Decision-Making  Simulation  (TDS),  on  the  other  hand,  has  only  recently  been  introduced 
in  some  of  these  schools.  There  is  very  little  knowledge  about  how  to  implement  it,  what 
it  can  accomplish,  and  whether  or  not  it  can  replace  or  augment  the  current  TDG  training 
regimen. 

To  evaluate  the  three  characteristics  mentioned  above,  we  will  use  two  different 
training  methodologies:  TDG  and  TDS. 

•  Ho  (Null  Hypothesis)  =  There  is  no  difference  between  the  abilities  of  the 
TDG  and  TDS  to  train  and  evaluate  military  personnel  with  respect  to 
decision-making,  leadership,  and  situational  awareness. 

•  Ha  (Alternative  Hypothesis)  =  There  is  some  difference  between  the  TDG 
and  TDS  training  methodologies  with  respect  to  their  ability  to  train  and 
evaluate  decision-making,  leadership,  and  situational  awareness  in 
military  personnel. 
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We  chose  an  a=  0.05  as  the  significance  criteria  whether  to  reject  the  null 
hypothesis.  A  2-tailed  t-test  will  be  used  to  determine  if  the  two  training  methods  are 
different  with  respect  to  evaluating/training  leadership,  decision-making,  and  situational 
awareness. 

2.  Procedure 

From  Experiment  1,  we  learned  lessons  which  we  implemented  in  this  study.  The 
design  of  this  experiment  remained  between-subjects.  Additionally,  we  realized  that  a 
single  training  session  would  not  be  adequate  to  build  the  necessary  skill  level.  Also,  it 
would  not  be  sufficient  for  participants  to  demonstrate  characteristics  of  any  of  the 
dependent  variables  (Leadership,  Decision-Making,  and  Situational  Awareness).  We 
decided  on  a  single  introductory  session  lasting  approximately  one  hour  to  teach  the  TDS 
participants  how  to  navigate  and  participate  in  a  typical  CCM  scenario.  The  same 
amount  of  time  was  reserved  for  the  TDG  group.  It  was  used  to  instruct  them  about  a 
TDG  session  and  how  to  participate  in  one.  Infantry  tactics,  techniques,  and  procedures 
(TTP)  instruction  was  provided  to  both  groups  during  this  first  hour  (same  instruction  for 
both  groups).  We  were  aware  that  unless  participants  had  prior  training  in  infantry  TTPs, 
they  would  only  be  able  to  understand  TTP  instruction  at  the  most  basic  level.  However, 
this  level  was  all  that  was  required  for  them  to  participate  effectively  in  either  the  TDG  or 
the  TDS  groups. 

We  made  every  effort  to  standardize  the  make-up  of  the  teams  in  the  TDG  and  the 
TDS.  The  TDG  and  the  TDS  groups  consisted  of  a  total  of  eight  personnel  each.  Within 
both  groups,  there  were  two  subgroups  of  four  participants  each.  In  both  TDG 
subgroups,  we  assigned  one  of  the  four  to  act  as  the  platoon  leader;  with  the  remaining 
three  to  act  as  squad  leaders.  Likewise,  in  the  TDS  subgroups,  there  was  one  platoon 
leader  and  three  squad  leaders.  All  participants  functioned  in  their  assigned  roles 
throughout  their  sessions. 

The  remaining  four  sessions  were  split  evenly  between  offensive  and  defensive 
scenarios.  The  CCM  software  comes  with  a  number  of  prepackaged  training  scenarios 
that  allow  for  individual  or  team-on-team  modes.  The  team-on-team  modality  was 
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selected  for  use  in  all  of  the  remaining  sessions  (Sessions  2-5).  The  same  built-in  CCM 
scenarios  were  used  by  the  TDG  group  to  ensure  that  the  TDS  and  TDG  groups  were 
presented  with  the  same  level  of  difficulty  within  their  scenarios.  The  CCM  scenarios 
changed  from  session  to  session  which,  consequently,  altered  the  troop  mix  and  terrain. 
Table  3  identifies  which  CCM  scenario  was  used  in  each  session  and  what  terrain  was 
presented  to  the  participants. 


Table  3.  TDG/TDS  Sessions  and  Scenarios 


Day 

CCM  Scenario 

Session 

TDG 

TDS 

Team  1 

Team  2 

Team  1 

Team  2 

1-la  Town 

1 

Introduction 

Introduction 

Introduction 

Introduction 

5-lb  Town 

2 

Defense 

Defense 

Offense 

Defense 

5-lb  Town 

3 

Offense 

Offense 

Defense 

Offense  j 

2-lb  Airfield 

4 

Defense 

Defense 

Offense 

Defense 

6- lb  Farm 

5 

Offense 

Offense 

Defense 

Offense 

We  collected  two  types  of  data:  self-reported  through  questionnaires  (pre  and 
post-session)  and  evaluations  by  the  instructors.  Once  each  session  had  concluded,  the 
participants  were  presented  with  questionnaires.  The  collected  data  was  used  later  to 
determine  relative  leadership,  decision-making  and  situational  awareness  scores.  Most  of 
these  scores  were  perceived  values  based  on  the  subjective  assessments  of  each 
participant  and  his/her  opinion  of  teammates’  performances  within  each  subgroup. 
Additionally,  the  instructors  evaluated  the  platoon  leaders  in  each  session  from  both  the 
TDG  and  TDS  groups.  It  is  clear  that  situational  awareness  can  only  be  evaluated  for  the 
TDS  participants.  However,  based  on  TDG  participants’  comments  and  discussions 
during  sessions,  we  expected  them  to  have  a  level  of  understanding  about  their 
teammates’  situational  awareness.  We  extracted  those  SA  answers  from  the 
questionnaires  and  used  them  to  determine  individual  SA  for  the  TDG  group  members. 

The  procedure  we  used  to  determine  SA  for  each  participant  was  called  the 
Geographical  Recall  and  Analysis  of  Data  in  the  Environment  (GRADE)  (Miller  and 
Shattuck  (2007)).  An  example  of  this  tool  can  be  found  in  Appendix  M.  In  the  middle  of 
the  TDS  sessions,  we  paused  the  simulation  and  evaluated  each  participant’s  knowledge 
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concerning  the  location,  size,  and  direction  of  movement  of  the  friendly  and  enemy 
forces.  We  did  this  by  having  them  draw  circles  and  arrows  on  a  map  sheet.  We  walked 
between  the  rooms  recording  the  exact  location,  size,  and  direction  of  troop  movement 
and  used  this  as  ground  truth.  All  TDS  participant  GRADE  sheets  were  evaluated  based 
on  this  ground  truth  sheet.  Use  of  this  method  was  not  possible  for  the  TDG  group 
because  the  TDG  offers  no  ground  truth  to  which  a  comparison  can  be  made. 

3.  Method 

a.  Participants 

The  participants  in  this  study  were  primarily  students  of  the  Naval 
Postgraduate  School  (N=16):  (3)  female  and  (13)  male,  ranging  from  21  to  43  years  old. 
Within  the  group  of  16,  there  were  two  Cadets  from  the  Air  Force  Academy  and  one 
from  the  U.S.  Military  Academy.  The  remaining  participants  were  from  three  different 
academic  curricula:  0399  (Modeling  Virtual  Environments  and  Simulation),  0365 
(Operations  Research),  and  0360  (Human  Systems  Integration).  The  HSI  program  is 
considered  non-technical  and  the  OR  and  MOVES  degrees  are  technical.  The  others 
were  military  officers  ranging  in  pay  grade  from  01-05.  Of  these,  there  were  officers 
from  the  U.S.  Navy,  U.S.  Army,  U.S.  Marine  Corps,  and  the  German  Navy.  Participants 
were  randomly  assigned  to  the  two  primary  groups  (TDG/TDS).  These  groups  did  not 
interact  throughout  the  experiment.  Additionally,  the  TDS  and  TDG  groups  were 
subdivided  into  two  smaller  internal  groups  (TDS  Group  A  &  B,  TDG  Group  A  &  B). 
We  collected  other  demographic  information:  age;  military  occupational  specialty; 
computer,  gaming,  and  simulation  experience;  total  service  time;  familiarity  with  infantry 
tactics,  techniques,  and  procedures;  service  component;  and  country  of  citizenship.  The 
following  tables  provide  a  more  concise  view  of  our  participants’  demographics.  Table  4 
is  granular  and  provides  information  by  subject  number  on  rank,  age,  time  in  service  in 
years,  and  Computer  Experience  in  hours/year. 
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Table  4.  Individual  Summary  Demographics 


General  Demographics 

Subject  # 

Rank 

Age 

TIS  Years 

Computer 

Experience 

TDG 

8 

03 

27 

5 

2920 

746 

03 

32 

4 

1460 

547 

Cadet 

22 

3 

3650 

261 

04 

36 

14 

1095 

432 

04 

37 

14.33 

365 

663 

Cadet 

21 

3 

1825 

517 

Cadet 

21 

3 

1460 

97 

02 

26 

4 

4380 

TDS  | 

733 

04 

34 

11 

1095 

377 

05 

43 

22.92 

2190 

343 

01 

23 

0.92 

1825 

64 

03 

37 

14.67 

1095 

967 

03 

30 

7.92 

2190 

784 

03 

31 

8 

365 

13 

03 

32 

8.58 

730 

537 

03 

35 

15.92 

1825 

In  addition  to  these  individual  demographics,  Tables  5,  6,  &  7  summarize 
the  participants  by  TDG  and  TDS  Group. 

Table  5.  Group  Summary  Demographics 


Age 

Time  In  Service 

Use  of  Personal  Computer 

TDG  Group 
(Years) 

TDS  Group 
(Years) 

TDG  Group 
(Years) 

TDS  Group 
(Years) 

TDG  Group 
(Hours/Year) 

TDS  Group 
(Hours/Year) 

N 

8 

8 

8 

8 

8 

8  ! 

Min 

21 

23 

3 

0.92 

365 

365 

Mean 

27.75 

33.13 

6.29 

11.24 

2144.38 

1414.38 

Median 

26.5 

33 

4 

9.79 

1642.5 

1460 

Max 

37 

43 

14.33 

22.92 

4380 

2190 

Std  Dev 

6.54 

5.79 

4.91 

6.6 

1371.79 

688.06 

We  can  see  in  Table  5  that  the  computer  use  among  the  TDG  group  was 
higher  than  the  TDS  group.  The  TDS  group  had  more  time  in  service.  This,  also, 
indicates  that  they  were,  on  average,  older  than  the  TDG  group. 
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Table  6.  Participants  with  Prior  Commercial  Video  Game  Experience 


INFORMATION  ASKED 

TDS 

TDG 

ALL 

Number  of  subjects  having  prior  experience  playing 
commercial  video  games 

7 

8 

15 

Number  of  subjects  NOT  having  prior  experience 
playing  commercial  video  games 

0 

0 

0 

Number  of  people  who  did  not  respond  to  this  question 

1 

0 

1 

We  can  see  in  Table  6  that  every  participant  has  had  some  level  of 
experience  playing  commercial  video  games. 

Table  7.  Summary  of  Commercial  Video  Game  Experience 


Subjects  Having  Prior  Experience  Playing  Commercial  Video  Games 

Type  of  Game 

TDS 

TDG 

ALL 

First  person  shooter 

Subjects  Responding 

5 

4 

9 

Total  #  of  Hours/ Year 

132 

76 

208 

Minimum 

2 

2 

2 

Maximum 

96 

52 

96 

Mean 

26.40 

19 

22.70 

Standard  Deviation 

39.63 

22.42 

31.45 

Flight  Simulation 

Subjects  Responding 

1 

1 

2 

Total  #  of  Hours/ Year 

96 

1 

97 

Minimum 

96 

1 

1 

Maximum 

96 

1 

96 

Mean 

96 

1 

48.50 

Standard  Deviation 

0 

0 

67.18 

Racing 

Subjects  Responding 

3 

0 

3 

Total  #  of  Hours/ Year 

23 

0 

23 

Minimum 

1 

0 

1 

Maximum 

20 

0 

20 

Mean 

7.67 

0 

7.67 

Standard  Deviation 

10.69 

0 

10.69 

Sports  (Football,  etc.) 

Subjects  Responding 

5 

2 

7 

Total  #  of  Hours/ Year 

367 

27 

394 

Minimum 

2 

3 

2 

Maximum 

240 

24 

240 

Mean 

73.40 

13.50 

43.45 
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Subjects  Having  Prior  Experience  Playing  Commercial  Video  Games 

Type  of  Game 

TDS 

TDG 

ALL 

Standard  Deviation 

100.60 

14.85 

87.40 

Puzzle,  Card,  Board 

Subjects  Responding 

5 

3 

8 

Total  #  of  Hours/ Year 

805 

774 

1579 

Minimum 

3 

2 

2 

Maximum 

730 

760 

760 

Mean 

161 

258 

209.50 

Standard  Deviation 

318.55 

434.77 

338.40 

Strategy 

Subjects  Responding 

3 

1 

4 

Total  #  of  Hours/ Year 

200 

12 

212 

Minimum 

24 

12 

12 

Maximum 

104 

12 

104 

Mean 

66.67 

12 

39.34 

Standard  Deviation 

40.27 

0 

42.76 

Adventure,  Fantasy 

Subjects  Responding 

3 

3 

6 

Total  #  of  Hours/ Year 

41 

1119 

1160 

Minimum 

5 

12 

5 

Maximum 

24 

1095 

1095 

Mean 

13.67 

373 

193.34 

Standard  Deviation 

9.61 

625.27 

441.77 

Arcade 

Subjects  Responding 

2 

0 

2 

Total  #  of  Hours/ Year 

97 

0 

97 

Minimum 

1 

0 

0 

Maximum 

96 

0 

96 

Mean 

48.50 

0 

48.50 

Standard  Deviation 

67.18 

0 

67.18 

Other 

Subjects  Responding 

0 

1 

1 

Total  #  of  Hours/ Year 

0 

1 

1 

Minimum 

0 

1 

0 

Maximum 

0 

1 

1 

Mean 

0 

1 

0.5 

Standard  Deviation 

0 

0 

0 

Table  7  shows  that  participants  had  the  most  commercial  game  experience 
with  puzzle,  card,  board,  and  adventure/fantasy  type  games.  Participants  demonstrated 
the  least  experience  with  the  racing  type  games. 
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Participants  were  all  treated  in  accordance  with  ethical  standards 
established  by  the  American  Psychological  Association.  Our  study  design  was,  also, 
approved  by  the  IRB  committee  at  NPS. 

We  performed  the  role  of  instructors  by  providing  the  introductory 
training  for  the  study,  including  the  infantry  TTP  training  in  Session  1.  For  Sessions  2-5, 
we  took  notes  on  performance,  answered  questions,  and  administered  the  surveys  before 
and  after  each  session.  Additionally,  we  found,  early  in  session  1  that  managing  a  study 
of  this  size  between  two  separate  rooms  was  quite  difficult  for  only  two  people.  Thus,  we 
enlisted  the  assistance  of  a  U.S.  Marine  Corps  student  in  the  Operations  Research 
curriculum  to  help  with  the  study’s  administrative  tasks. 

b.  Apparatus 

The  lab  space  was  purposefully  configured  in  a  way  to  support  the 
isolation  of  the  two  internal  TDG  and  TDS  subgroups.  Each  subgroup  occupied  either 
Room  A  or  Room  B  as  seen  in  Figure  6. 


Figure  6.  Experiment  2  -  Lab  Setup 

Each  squad  leader  was  provided  with  a  Dell  Pentium-4M  laptop  computer 
(6  all  together)  with  CCM  installed  on  it.  The  squad  leaders  used  it  to  execute  their  own 
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orders,  as  well  as,  to  facilitate  the  commands  of  the  platoon  leader.  These  laptops  were 
part  of  a  closed  experimentation  network.  This  network  was  the  basis  for  the  CCM 
virtual  environment  that  allowed  six  users  (three  offense  and  three  defense)  to  interact 
with  each  other  relative  to  the  CCM  scenario  for  that  session.  The  environment 
computers  were  connected  with  a  16-port  Ethernet  Switch  using  Cat  5  UTP  cabling. 

For  administrative  purposes  and  training,  an  IBM  laptop  and  an  overhead 
projector  were  used  to  demonstrate  infantry  TTPs  and  the  procedures  for  participating  in 
both  the  TDG  and  TDS  sessions.  We  used  two  digital  camcorders  (audio/visual),  one  in 
each  room,  to  record  participant  activities  during  the  sessions.  We  had  multiple  copies  of 
a  Close  Combat  Marines  Memory  Sheet  (see  Appendix  D),  the  operations  order  relevant 
to  the  particular  scenario  (see  Appendix  K),  and  memory  sheet  for  the  offensive  and 
defensive  infantry  tactics  (see  Appendix  L)  presented  in  the  first  session.  There  were 
white  boards  in  both  rooms  for  the  subgroups  to  diagram  their  plans  and  briefs  during 
their  group  discussion  time.  We  provided  scratch  paper  and  pens  on  all  tables  for  similar 
planning  purposes.  We  used  a  twelve-sided  die  to  randomly  assign  the  participants  to  the 
groups.  This  die  was,  also,  used  to  assign  a  unique  identifier  to  each  participant. 

c.  Procedure 

From  within  the  NPS  student  population,  we  acquired  a  total  of  16 
participants.  Then,  we  divided  them  into  either  a  Tactical  Decision-Making  Game  (TDG) 
group  or  a  Tactical  Decision-Making  Simulation  (TDS)  group.  The  TDS  group  used 
Close  Combat  Marines  (CCM).  The  TDG  and  TDS  groups  were  further  divided  into  two 
subgroups  containing  four  subjects  each.  The  TDG  and  TDS  groups  participated  in  the 
study  at  different  times  and  on  different  days.  See  Table  8. 
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Table  8.  Day  and  Time  for  TDG  &  TDS  Sessions 


Week  1 

Week  2 

Thursday 

Friday 

Saturday 

Sunday 

Monday 

Tuesday 

Wednesday 

0800-0900 

0900-1000 

1000-1100 

1100-1200 

1200-1300 

TDS 

1300-1400 

Sessions  3  &  4 

TDG 

TDG 

1400-1500 

TDG 

Sessions  3  &  4 

Session  5 

1500-1600 

TDS 

Sessions  1  &  2 

TDS 

1600-1700 

Session  1  &  2 

Session  5 

We  chose  different  CCM  battle  scenarios  in  the  different  TDG/TDS 
sessions.  To  ensure  a  strong  basis  for  comparison,  our  intent  was  to  ensure  that  both 
groups  received  an  equal  number  of  scenarios.  However,  the  TDS  group  mandated  that 
one  subgroup  be  in  the  offense  while  the  other  was  in  the  defense.  This  was  because  the 
teams  were  fighting  against  one  another  in  the  virtual  environment.  On  the  other  hand, 
the  TDG  group  had  no  virtual  environment.  Thus,  we  controlled  the  nature  of  the 
scenario  across  each  session  (both  subgroups  were  either  on  offense  or  defense).  For 
example,  the  TDG  Session  2  was  defensive  for  both  TDG  subgroups  and  Session  3  was 
offensive  for  both  TDG  subgroups.  The  operation  order  and  overall  scenarios  used  in  the 
TDG  were  drawn  from  the  TDS  CCM. 

(1)  Tactical  Decision-Making  Game.  The  five  TDG  sessions 
consisted  of  one  introductory  session  and  four  training  sessions.  In  the  first  session,  all 
TDG  participants  were  provided  with  instruction  on  basic  infantry  offensive  and 
defensive  tactics,  techniques,  and  procedures.  We  explained  to  them  the  exact  nature  of 
the  TDG,  how  to  participate  in  one,  and  the  exact  planning  sequence  they  needed  follow 
to  complete  the  TDG  effectively.  The  memory  sheets  for  both  the  TTPs  and  the 
suggested  planning  process  were  available  to  participants  and  were  distributed  onto  the 
tables  for  both  groups. 

At  the  beginning  of  each  TDG  session  (Sessions  2-5),  we  read  the 
operation  order  to  the  entire  cohort  in  Room  B  (see  Figure  6).  The  groups  were  divided 
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into  Group  1  and  2.  Group  1  remained  in  Room  B,  while  Group  2  moved  to  Room  A  for 
planning  time.  Each  group  discussed  the  operations  order  internally  and,  then,  turned  to 
individual  planning.  Each  individual  planned  his  response  to  the  operation  order  and 
formulated  his  best  effort  at  a  course  of  action.  After  this  phase,  the  individuals  came 
back  together  for  a  group  planning  session.  During  this  time,  a  leader  was  randomly 
selected  to  direct  his  team  and  to  establish  the  overall  team  plan.  Once  each  TDG 
subgroup  had  its  team’s  plan  completed,  Group  2  moved  from  Room  B  to  Room  A. 
Thus,  the  entire  TDG  group  (8  persons  in  all)  gathered  in  one  room  for  an  After  Action 
Review  (AAR).  Each  team  leader,  in  turn,  briefed  his  team’s  plan  to  the  entire  TDG 
group.  See  Figure  7. 


Figure  7.  TDG  Scenario  Underway 


The  instructors  and  all  participants  were  free  to  comment  on  and 
ask  questions  regarding  the  nature  of  the  plans.  This  applied  to  why  the  team  decided  on 
a  particular  course  of  action.  After  this  phase  concluded,  each  TDG  participant  filled  out 
a  questionnaire  related  to  the  session  that  he  had  just  finished.  Additionally,  the 
instructors  evaluated  the  platoon  leader’s  leadership  and  decision-making  performance  on 
a  separate  evaluation  form.  The  same  process  was  repeated  for  the  remaining  three  TDG 
sessions. 
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(2)  Tactical  Decision-Making  Simulation.  Upon  arriving  at  the 
lab,  the  participants  were  divided  into  two  TDS  subgroups.  The  initial  training  session 
was  the  same  for  the  TDS  group  as  for  the  TDG  group  except  for  the  hands-on  CCM 
training.  The  two  subgroups  (each  had  four  participants)  played  against  each  other  in  the 
remaining  Sessions  2-5.  This  mandated  that  one  group  be  in  the  offensive  role  and  the 
other  group  in  the  defensive.  Since  the  operation  orders  were  different  for  each  group 
and  knowledge  about  the  each  other’s  orders  could  have  potentially  compromised  critical 
intelligence,  the  subgroups  were  immediately  separated  so  they  would  not  be  able  to  hear 
each  other’s  internal  discussions.  The  subgroups  remained  separated  in  different  rooms 
throughout  the  entire  session  (See  Figure  6.  Lab  Space  Layout).  This  technique  ensured 
that  each  team  operated  independently  and  did  not  have  prior  knowledge  of  their 
opponent’s  plan. 

A  platoon  leader  for  each  TDS  subgroup  was  randomly  assigned  at 
the  beginning  of  each  session.  That  person  acted  as  the  leader  for  not  only  the  planning 
session,  but  for  the  conduct  of  the  entire  TDS  scenario  for  his  subgroup.  The  groups 
were  given  approximately  3-5  minutes  of  planning  time  before  they  actually  started 
CCM.  Once  CCM  had  started,  the  groups  had  a  short  time  (2  minutes)  to  place  their 
troops  in  the  battle  space.  At  this  time,  we  started  the  actual  scenario  where  each  team 
began  to  implement  their  platoon  leader’s  plan  within  the  virtual  environment.  Each 
subgroup  consisted  of  three  squad  leaders  sitting  at  computer  terminals  while  the  platoon 
leader  stood  behind  them,  issued  orders,  and  directed  the  overall  team  action.  The 
internal  CCM  system  algorithms  determined  when  to  end  the  session  based  on  the 
number  of  victory  locations  that  each  group  controlled  and  their  number  of  troop  losses. 
In  all  sessions,  the  scenarios  were  allowed  to  go  to  completion. 
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Figure  8.  TDS  Scenarios  Underway 


We  monitored  the  activities  of  the  TDS  subgroups  seen  in  Figure 
8.  One  researcher  was  with  Group  1  in  Room  A  and  the  other  was  with  Group  2  in  Room 
B.  Additionally,  we  acquired  the  assistance  of  an  U.S.  Marine  Corps  infantry  officer. 
This  person  assisted  us  with  the  administration  of  the  experiment,  advice  on  infantry 
TTPs,  and  the  discussion  in  the  after  action  review  (AAR)  session  for  both  the  TDG  and 
the  TDS.  (A  detailed  minute-by-minute  schedule  of  the  TDG  and  TDS  sessions  can  be 
found  in  Appendix  J). 


After  each  TDS  session,  each  participant  filled  out  a  questionnaire 
and  entered  his/her  opinions  regarding  how  their  teammates  performed  on  decision¬ 
making,  situational  awareness,  and  leadership.  We  evaluated  the  platoon  leaders  on 
decision-making  and  leadership.  Additionally,  we  used  GRADE  (Shattuck  and  Miller, 
2007)  to  evaluate  each  participant’s  situational  awareness  at  two  points  during  each 
session  (2-5).  However,  we  were  not  able  to  accomplish  this  as  we  intended.  There  were 
certain  days  where  TDS  sessions  ended  earlier  than  expected  and  prior  to  the  full  40 
minutes  allotted.  In  cases  where  this  occurred,  we  had  only  one  GRADE  worksheet  (see 
Appendix  M)  from  the  participants  to  evaluate  their  SA  for  that  session. 


4.  Results 

This  section  lists  results  from  two  different  types  of  analyses  from  Experiment  2. 
The  longitudinal  analysis  was  descriptive  and  examined  each  participant’s  scores  on 
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leadership,  decision-making,  and  SA  across  all  sessions.  The  inferential  analysis 
conducted  in  sections  b,  c,  and  d.  examined  correlations  among  various  factors  of 
interest. 


a.  Longitudinal  Analysis  over  All  TDG/TDS  Sessions 

For  the  purpose  of  preserving  our  participants’  anonymity,  we  assigned 
them  random  subject  identifiers  which  can  be  seen  in  all  of  the  longitudinal  graphs.  We 
randomized  their  assignment  to  groups  and  their  participant  numbers  to  prevent  one 
participant  thinking  he  was  better  than  another  participant  was,  i.e.,  that  a  person  with  the 
number  1  would  be  ranked  higher  than  a  person  with  the  number  2. 

The  TDS  leaders  were  appointed  early  in  the  scenario  while  the  TDG 
leaders  were  appointed  in  the  middle  of  the  session  only  after  individual  planning  had 
been  completed.  For  a  detailed  minute-by-minute  list  of  the  TDG  and  TDS  scenarios,  see 
Appendix  J  -  TDG/TDS  Schedule  (Experiment  2). 

(1)  Leadership.  For  the  leadership  evaluations,  each 
participant  graded  his  entire  team  as  a  percentage  of  100  (the  total  100  points  was  divided 
among  all  team  members).  This  grading  method  was  present  for  only  the  leadership 
scores.  The  other  dependent  variables  (decision-making  and  situational  awareness)  were 
graded  on  a  Likert  scale  1-7.  The  leadership  data  for  the  TDG  and  TDS  participants 
revealed  that  the  designated  leader  emerged  as  the  perceived  leader  among  all  other 
participants  without  exception. 

Figure  9  summarizes  the  results  from  the  leadership  evaluation 
between  the  TDG  and  TDS  groups. 
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TDG  Team  A  -  Leadership 


TDG  Team  B  -  Leadership 
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TDS  Team  A  -  Leadership 


TDS  Team  A  -  Leadership 


Figure  9.  Longitudinal  Results  -  Leadership  (Self-Reported) 


Figure  9  shows  that  each  TDS  participant’s  leadership  scores 
fluctuated  greatly  between  the  sessions  in  which  he/she  acted  as  platoon  leader  versus 
those  as  a  squad  leader.  The  TDS  scores  had  a  much  higher  deviation  from  the  mean 
than  the  TDG  group.  This  could  be  attributed  to  the  more  active  role  played  by  the  leader 
in  the  TDS  group.  The  TDS  platoon  leader  was  generally  standing  behind  his  squad 
leaders  directing  their  action  only  when  they  were  actively  playing  CCM.  No  such 
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activity  was  present  in  the  TDG  due  to  the  nature  of  the  TDG.  In  the  TDS,  though,  the 
leaders  were  required  to  act,  give  orders,  make  changes  to  their  plan,  and  suffered 
consequences  of  inaction  which  resulted  in  troop  losses  and  attacks  by  a  thinking  enemy. 
These  activities  appeared  to  propel  the  TDS  leader  to  a  higher  level  of  performance  —  a 
fact  born  out  by  the  results  of  the  overall  scoring  by  his  subordinate  squad  leaders. 

(2)  Decision-Making.  The  overall  decision-making  trend  for 
the  TDG  group  was  relatively  flat  as  seen  in  Figure  10.  The  TDS  teams  were  either 
overall  increasing  for  Team  A  or  decreasing  for  Team  B.  Participant  number  343  in  the 
TDS  Team  A  was  absent  from  Sessions  3  and  4  and,  therefore,  received  no  score  (a  score 
of  zero).  Interestingly,  this  trend  of  teammates  grading  one  another  higher  in  decision¬ 
making,  when  they  won  the  scenario,  and  lower,  when  they  lost,  corresponded  to  the 
win/loss  results  and  spanned  all  sessions  for  TDS  teams.  Team  A  was  defeated 
decisively  in  Sessions  2,  3,  and  4,  but  made  improvements  in  their  ability  to  work 
together  and  in  their  decision-making  ability.  Team  B’s  performance  decreased  over  the 
course  of  all  sessions.  Even  though  they  won  all  but  the  last  session,  Team  A  continued 
to  improve  at  a  faster  rate  than  Team  B.  They  were  defeated  in  the  last  session  due  to  the 
superior  firepower  (availability  of  artillery)  used  by  their  opponent.  These  trends  could 
indicate  that  the  team  members  were  equating  their  ability  to  make  decisions  with  the 
outcome  of  the  sessions. 

Even  though  all  teams  were  randomly  assigned,  three  of  the  four 
members  of  TDS  Team  B  had  been  classmates  in  previous  NPS  courses  and  are  now  all 
in  the  same  Human  Systems  Integration  curriculum.  The  graph  of  TDS  Team  B  could 
indicate  an  initial  overconfidence  in  their  ability  to  work  together.  However,  this 
decreased  over  time  as  the  other  team  became  more  successful.  The  graph  of  TDS  Team 
A  -  Decision-Making  could  indicate  that  the  group  was  coalescing  as  a  team  throughout 
all  sessions.  They  worked  better  together  and  were  able  to  make  better  decisions  as  the 
sessions  progressed. 

Figures  10  and  1 1  demonstrate  a  consistency  between  the  sessions 
on  decision-making  and  situational  awareness.  As  stated  earlier,  participant  343  of  TDS 


60 


Team  A  received  a  score  of  zero  in  Sessions  3  and  4.  This  was  a  result  of  343 ’s  absence 
from  the  experiment  during  those  two  sessions. 
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TDS  Team  B  -  Decision-Making 


Figure  10.  Longitudinal  Results  -  Decision-Making  (Self-Reported) 


(3)  Situational  Awareness.  The  graphs  in  Figures  10  and  11  are 
almost  identical  for  both  TDG  and  TDS  groups.  The  participants’  scores  indicate  that  the 
TDG  and  TDS  groups’  situational  awareness  scores  over  all  sessions  correspond  strongly 
to  their  decision-making  scores. 
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Figure  11  provides  a  basis  for  comparing  SA  between  the  two 
groups  (TDG  &  TDS). 


TDG  Team  A  -  Situational  Awareness 


TDG  Team  B  -  Situational  Awareness 


TDS  Team  A  -  Situational  Awareness 


TDS  Team  B  -  Situational  Awareness 


Figure  1 1 .  Longitudinal  Results  -  Situational  Awareness  (Self-Reported) 


The  previous  longitudinal  graphs  show  the  overall  results  for 
leadership,  decision-making,  and  situational  awareness  for  all  participants  in  the  TDG 
and  TDS  groups  across  all  sessions. 
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During  this  study,  we  were  concerned  with  several  more 
parameters  beside  the  leadership,  decision-making,  and  situational  awareness  skills.  We 
were,  also,  interested  in  how  other  key  skills  required  of  a  platoon  level  leader  could  be 
improved  through  the  use  of  the  TDG  and  TDS.  Figures  12  and  13  show  the  absolute  and 
relative  improvement  within  the  two  training  groups  (TDG  and  TDS)  from  the  beginning 
session  to  the  final  session.  We  drew  this  information  from  questions  4  and  5  in  the  final 
TDG  questionnaire  and  questions  14  and  15  in  the  final  TDS  questionnaire.  The  scores 
reflected  in  Figures  12  and  13  are  averages  for  all  participants  in  the  TDG  and  in  the  TDS 
groups.  They,  also,  show  that  there  was  a  perceived  improvement  across  all  domains. 
Figures  12  and  13  demonstrate  the  difference  between  the  absolute  scores  at  the 
beginning  of  Session  1  and  the  absolute  and  relative  improvement  reported  in  the  final 
questionnaire  after  the  final  session. 

Y 

+  Initial  Absolute  Score 
X  Final  Session  Relative  Improvement 
□  Final  Session  Absolute  Score 


Figure  12.  TDG  Absolute  and  Relative  Improvement 
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In  Figure  12,  we  can  see  that  participants  stated  that  they  showed 
improvement  over  all  domains  because  each  line  has  some  measurable  length.  Since  the 
TDG  is  a  pen  and  paper  drill  with  no  simulation  involved,  participants  only  had  the 
ability  to  engage  in  the  scenario  within  their  own  mind.  They  never  saw  it  played  out  in 
an  objective  manner.  We  believe  that  this  was  the  reason  why  the  “Tactical  Movement  to 
an  Objective”  and  the  “Fire  &  Maneuver”  tasks  showed  the  least  amount  of 
improvement.  However,  overall  perception  of  absolute  and  relative  improvement  across 
all  domains  was  small.  Although  TDG  and  TDS  subjects  participated  equally  in 
offensive  and  defensive  scenarios,  the  TDG  group  believed  that  they  improved  more  on 
the  defensive  tasks.  In  Figure  13,  we  can  see  that  the  magnitude  of  the  improvements  in 
the  TDS  group  was  greater  than  in  the  TDG  group.  However,  this  was  not  by  a 
significant  margin. 
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Figure  13.  TDS  Absolute  and  Relative  Improvement 
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In  Figure  13,  TDS  participants  stated  that  they  felt  they  improved 
over  all  domains,  though  some  more  than  others.  We  noticed  that  in  the  “Issue  Order” 
domain,  their  relative  improvement  was  far  less  than  their  absolute  score.  The  domains 
“Tactical  Movement  in  the  Offense”  and  “Tactical  Movement  Objective”,  also, 
demonstrated  this  same  condition,  but  was  by  an  insignificant  amount.  We  believe  that 
the  minimal  improvement  on  the  “Issue  an  Operation  Order”  task  was  because 
participants  did  not  issue  operation  orders;  rather,  the  orders  were  only  provided  for  them 
in  written  form.  The  TDS  participants,  also,  practiced  offensive  and  defensive  tactics 
equally.  However,  they  thought  that  they  improved  most  in  offensive  operations.  This  is 
in  direct  opposition  to  the  TDG  group  results.  We  believe  that  the  TDG  format  implied, 
to  the  students,  that  they  were  in  more  of  a  defensive  role,  while  the  TDS  format  implied 
more  of  an  offensive  role. 

b.  Hypothesis  Testing  -  TDG/TDS  Participant  Scores 

Since  our  experiment  was  about  testing  and  evaluating  leadership, 
decision-making,  and  situational  awareness,  we  decided  to  examine  them  based  on 
differences  between  the  TDG  and  the  TDS  groups.  A  t-test  was  used  because  there  were 
only  two  different  experimental  groups.  Since  there  was  no  good  reason  to  believe  that 
the  scores  would  fall  in  a  particular  direction,  we  decided  to  use  a  2-tailed  t-test  to 
evaluate  the  variables.  For  all  hypothesis  tests,  we  chose  the  a  level  =  0.05  to  be 
significance  level  controlling  whether  to  reject  the  null  hypothesis  or  not. 

(1)  Peer  Evaluation. 

Leadership.  The  participants’  peer  leadership  scores  were  taken 
from  question  2  on  questionnaires  from  Sessions  2-5.  Each  participant  received  a 
leadership  score  from  his  teammates  such  that  there  were  a  total  of  three  different  scores 
per  participant.  This  was  averaged  to  achieve  a  session  average  and  was  repeated  for  all 
sessions.  Finally,  all  participant  session  averages  were  averaged  again  to  find  the  total 
leadership  score  for  each  participant. 

Ho  (Null  Hypothesis)  =  There  is  no  difference  between  the  TDG 
and  TDS  participant  leadership  scores  given  by  their  peers. 
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Ha  (Research  Hypothesis)  =  There  is  a  difference  between  the 
TDG  and  TDS  participant  leadership  scores  given  by  their  peers. 

A  2-tailed  t-test  determined  p  =  0.8747  >  0.05.  We  do  not  reject 
the  null  hypothesis.  We  conclude  that  there  is  no  difference  between  the  participants’ 
scores  when  evaluated  by  their  peers  on  leadership  when  comparing  the  TDG  and  TDS 
groups.  This  suggests  that  the  TDG  and  the  TDS  provide  opportunity  to  exercise  and  to 
evaluate  leadership. 

Decision-Making.  The  participants’  peer  decision-making  scores 
were  taken  from  questions  8,  9,  and  12  on  questionnaires  from  sessions  2-5.  Each 
participant  received  a  decision-making  score  from  his  teammates  such  that  there  were  a 
total  of  three  different  scores  per  participant.  This  was  averaged  to  achieve  a  session 
average  and  was  repeated  for  all  sessions.  Finally,  all  participant  session  averages  were 
averaged  again  to  find  the  total  decision-making  score  for  each  participant. 

Ho  (Null  Hypothesis)  =  There  is  no  difference  between  the  TDG 
and  TDS  participant  decision-making  scores  given  by  their  peers. 

Ha  (Research  Hypothesis)  =  There  is  a  difference  between  the 
TDG  and  TDS  participant  decision-making  scores  given  by  their  peers. 

A  2-tailed  t-test  determined  p  =  0.0356  <  0.05.  We  do  not  reject 
the  null  hypothesis.  We  conclude  that  there  is  a  difference  between  the  TDG/TDS 
participants’  scores  on  the  peer  decision-making  evaluation.  Since  we  assigned  the 
participants  into  groups  randomly,  we  think  that  the  real  difference  was  caused  by  the 
method  used,  not  by  differences  in  how  the  peers  scored  each  other  within  the  groups. 

Situational  Awareness.  The  participants’  peer  Situational 
Awareness  scores  were  taken  from  questions  12  and  16-18  on  questionnaires  from 
Sessions  2-5.  Each  participant  received  a  situational  awareness  score  from  his  teammates 
such  that  there  were  a  total  of  three  different  scores  per  participant.  These  were  averaged 
to  achieve  a  session  average  and  were  repeated  for  all  sessions.  Finally,  all  participant 
session  averages  were  averaged  again  to  find  the  total  decision-making  score  for  each 
participant. 


66 


Ho  (Null  Hypothesis)  =  There  is  no  difference  between  the  TDG 
and  TDS  participant  Situational  Awareness  scores  given  by  their  peers. 

Ha  (Research  Hypothesis)  =  There  is  a  difference  between  the 
TDG  and  TDS  participant  Situational  Awareness  scores  given  by  their  peers. 

A  2-tailed  t-test  determined p  =  0.0068  <  0.05.  We  reject  the  null 
hypothesis  and  conclude  that  there  is  a  difference  between  the  participants’  scores  on  the 
peer  situational  awareness  evaluation  in  TDG  and  TDS.  Since  we  assigned  the 
participants  into  groups  randomly,  we  think  that  the  real  difference  is  caused  by  the 
method  used  and  not  by  differences  in  how  the  peers  in  the  groups  scored  one  another. 

(2)  Researcher  Evaluation. 

Leadership.  We,  also,  evaluated  the  TDG  and  TDS  participants 
with  respect  to  their  leadership.  Their  scores  were  taken  from  the  Researcher  Evaluation 
Form  (see  Appendix  N)  questions  15-16.  Each  participant  was  chosen  to  be  the  team 
leader  for  exactly  one  session.  This  was  the  only  session  for  which  we  evaluated  him  and 
obtained  a  leadership  score. 

Ho  =  (Null  Hypothesis)  There  is  no  difference  between  TDG  and 
TDS  participant  leadership  scores  given  by  the  researchers. 

Ha  =  (Research  Hypothesis)  There  is  a  difference  between  TDG 
and  TDS  participant  leadership  scores  given  by  the  researchers. 

A  2-tailed  t-test  determined  p  =  0.1765  >  0.05.  We  do  not  reject 
the  null  hypothesis.  We  conclude  that  there  is  no  difference  between  the  participants’ 
scores  on  researcher  leadership  evaluation  in  TDG  and  TDS. 

Decision-Making.  We  also  evaluated  the  TDG  and  TDS 
participants  with  respect  to  their  decision-making  ability.  Their  scores  were  taken  from 
the  Researcher  Evaluation  Form  (see  Appendix  N)  questions  13-14.  Each  participant  was 
chosen  to  be  the  team  leader  for  exactly  one  session.  This  is  the  only  session  for  which 
we  evaluated  him  and  obtained  a  leadership  score. 
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Ho  =  (Null  Hypothesis)  There  is  no  difference  between  the  TDG 
and  TDS  participant  decision-making  scores  given  by  the  researchers. 

Ha  =  (Research  Hypothesis)  There  is  a  difference  between  the 
TDG  and  TDS  participant  decision-making  scores  given  by  the  researchers. 

A  2-tailed  t-test  determined  p  =  0.1606  >  0.05.  We  do  not  reject 
the  null  hypothesis.  We  conclude  that  there  is  no  difference  between  the  participants’ 
scores  on  the  researcher  decision-making  evaluation  in  TDG  and  TDS. 

Situational  Awareness.  We  were  able  to  evaluate  the  TDS 
participants  on  their  SA,  but  not  for  the  TDG  participants.  The  TDS  group  had  a  CCM, 
which  maintained  a  state  of  all  enemy  and  friendly  unit  locations.  As  we  mentioned 
earlier,  we  drew  their  ground  truth  from  this  system  with  an  SA  instrument  called  the 
Geographical  Recall  and  Analysis  of  Data  in  the  Environment  (GRADE)  (Miller  & 
Shattuck,  2007).  We  compared  each  participant’s  GRADE  sheet  with  ground  truth  on  a 
Likert  scale  1-7.  Their  average  SA  score  was  a  4.19.  The  TDG  group  had  no  ground 
truth  from  which  to  draw  because  they  did  not  use  a  computer  system  that  maintained  a 
record  of  their  planning  process.  As  a  result,  their  SA  could  only  be  evaluated  based  on  a 
subjective  peer  assessment  from  the  questionnaires. 

(3)  Summary  of  Results.  Results  of  the  Hypothesis  tests  for 
leadership,  DM,  and  SA  are  located  in  the  tables  below.  Table  9  presents  the  summary  of 
the  peer  evaluations  and  Table  10  presents  the  instructors’  evaluations. 


Table  9.  Peer  Evaluation  t-test  Comparing  TDG/TDS 


P  Value 

Result 

Leadership 

p  =  0.8747  >  0.05 

No  significant  difference  between 
TDG/TDS  leadership  scores 

Decision  Making 

p  =  0.0356  <  0.05 

Significant  difference  between 
TDG/TDS  decision-making  scores 

Situational  Awareness 

p  =  0.0068  <  0.05 

Significant  difference  between 
TDG/TDS  SA  scores 
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Table  10.  Instructor  Evaluation  t-test  Comparing  TDG/TDS 


Leadership 

p  =  0.1765  >  0.05 

No  significant  difference  between 
TDG/TDS  leadership  scores 

Decision  Making 

p  =  0.1606  >0.05 

No  significant  difference  between 
TDG/TDS  leadership  scores 

Situational  Awareness 

*  No  Basis  for  Comparison  * 

The  TDG  was  a  mental  simulation  that  was  not  based  upon  a 


collective  understanding  of  ground  truth  for  the  scenario.  Each  person  had  his  own 
understanding  of  where  the  enemy  was  and  what  they  were  doing.  The  TDS  group  had 
ground  truth  established  from  within  the  computer  simulation.  Due  to  the  varying  nature 
of  these  two  methods  for  determining  situational  awareness,  there  was  no  basis  from 
which  to  compare  the  TDG  and  the  TDS  from  the  instructor’s  point  of  view. 

c.  TDG  Correlation  Analysis 

All  factors  were  scored  based  on  each  participant’s  subjective  perception 
of  the  questions  asked.  In  other  words,  if  the  participant  was  grading  his  peers  on 
leadership,  he  scored  them  based  on  his  own  opinions  and  understanding  of  what  a  good 
or  bad  leader  should  be.  This  was  true  for  all  factors  in  the  below  analysis. 

(1)  Factor:  Talkativeness  Predicts  Leadership. 

Questions 

•  Leadership  Question  (Sessions  2-5/Question  2)  -  Each 
participant  was  asked  to  provide  a  score  between  0  and  100 
for  each  of  their  team  members.  The  final  score  must  total 
100.  This  score  essentially  ranked  the  participants  with 
regard  to  their  perceived  leadership  ability  during  the 
sessions. 

•  Talkativeness  Question  (Sessions  2-5/Question  6)  -  The 
same  scoring  method  (between  0  and  100)  was  used  for  the 
amount  of  talkativeness  exhibited  by  the  participants  during 
the  sessions. 
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Results 


Figure  14  demonstrates  that  there  is  a  strong  linear  relationship 
between  talkativeness  and  the  perception  of  leadership. 


Regression  Plot 


Actual  by  Predicted  Plot 


Figure  14.  Graph  -  Talkativeness/Leadership 


The  P  value  for  the  test  for  significance  of  regression  is  P  =  0.0005 
and  RSqr  =  0.88.  This  clearly  indicates  strong  evidence  that  talkativeness  is  linearly 
related  to  leadership. 


(2)  Factor:  Age  Predicts  Leadership. 

Questions 

•  Age  Question.  (Session  1 /Question  9)  -  This  question 
asked  for  the  participant’s  year  of  birth.  The  age  was 
determined  by  subtracting  this  from  the  current  year. 

•  Leadership  Question.  (Sessions  2-5/Question  2)  -  see 
Factor:  Talkativeness  Predicts  Leadership. 

Results 


Figure  15  demonstrates  a  strong  correlation  between  age  and  the 
perception  of  leadership. 


70 


Regression  Plot 


Actual  and  Predicted  Plot 


Figure  15.  Graph  -  Age/Leadership 


The  P  value  for  the  test  for  significance  of  regression  is  P  =  0.0262 
and  RSqr  =  0.59.  The  regression  Model  indicates  that  Age  is  a  strong  predictor  of 
perceived  leadership  ability.  In  other  words,  the  older  the  participant  was  the  stronger 
leadership  score  he  received. 


Making. 


(3)  Factor:  Situational  Awareness  Predicts  Good  Decision- 


Questions 

•  Decision-Making  Question  (Sessions  2-5/Question  12: 
Items  8  &  9)  -  Participants  were  asked  about  their  opinion 
of  their  teammates  with  regard  to  their  ability  to  identify 
the  enemy’s  center  of  gravity  and  to  attack  it.  Also,  they 
were  asked  about  how  well  the  other  team  members 
adjusted  to  the  changing  enemy  situation.  All  scores  were 
graded  on  a  1-7  Likert  Scale. 

•  Situational  Awareness  Question.  (Sessions  2-5/Question 
12,  Items  16,  17,  &  18)  -  Participants  were  asked  about 
their  confidence  regarding  the  enemy  location,  enemy  size, 
and  direction  of  enemy  movement. 

Results 


Figure  16,  also,  demonstrates  that  SA  is  strongly  correlated  with 
the  ability  to  make  decision-making. 
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Regression  Plot 


Actual  by  Predicted  Plot 


Figure  16.  Graph  -  Situational  Awareness/Decision  Making 


The  P  value  for  the  test  for  significance  of  regression  is  P  =  0.0004 
and  RSqr  =  0.89.  This  clearly  indicates  strong  evidence  that  decision-making  is  linearly 
related  to  situational  awareness. 

(4)  Factor:  Correlation  between  the  Internal  Team  Members’ 
Scores  and  the  Instructors’  Score  on  Leadership. 

Questions 

See  above  Talkativeness  Predicts  Leadership.  This  measure 
examined  the  relationship  between  the  leadership  score  a  participant  received  from  his 
fellow  teammates  and  the  leadership  score  the  instructors  gave  him  during  his  role  as 
team  leader. 

Results 

Figure  17  shows  that  there  is  some  relationship  between  the 
instructor  leadership  score  and  the  leadership  score  that  the  participants  received  from 
their  peers. 
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Regression  Plot 
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Actual  by  Predicted  Plot 


Figure  17.  Graph  -  Instructor  Leadership/Perceived  Leadership 


The  P  value  for  the  test  for  significance  of  regression  is  P  =  0.0143 
and  RSqr  =  0.66.  This  clearly  indicates  that  there  is  evidence  that  the  instructor 
evaluation  of  the  participants’  leadership  ability  is  linearly  related  to  the  leadership 
evaluation  given  to  the  participant  by  his  peers. 

d.  TDS  Correlation  Analysis 

(1)  Factor:  Talkativeness  Predicts  Leadership. 

Questions 

•  Leadership  Question  (Sessions  2-5/Question  2)  -  Each 
participant  was  asked  to  provide  a  score  between  0  and  100 
for  each  of  their  team  members.  The  final  score  must  total 
100.  This  score  essentially  ranked  the  participants  with 
regard  to  their  perceived  leadership  ability  during  the 
sessions. 

•  Talkativeness  Question  (Sessions  2-5/Question  6)  -  The 
same  scoring  method  (between  0  and  100)  was  used  for  the 
amount  of  talkativeness  exhibited  by  the  participants  during 
the  sessions. 

Results 

Figure  18  demonstrates  a  weak  correlation  between  talkativeness 

and  leadership. 
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Regression  Plot 


Actual  by  Predicted  Plot 


Figure  18.  Graph  -  Talkativeness/Leadership 


The  P  value  for  the  test  for  significance  of  regression  is  P  =  0.005 
and  RSqr  =  0.48.  This  indicates  evidence  that  talkativeness  is  linearly  related  to 
leadership. 


(2)  Factor:  Age  Predicts  Leadership. 

Questions 

•  Age  Question.  (Session  1 /Question  9)  -  This  question 
simply  asked  for  the  participant’s  year  of  birth.  The  age 
was  determined  by  subtracting  this  from  the  current  year. 

•  Leadership  Question.  (Sessions  2-5/Question  2)  -  see 
Factor:  Talkativeness  Predicts  Leadership. 

Results 


The  regression  plot  below  in  Figure  19  demonstrates  that  age  is 
correlated  with  the  perceived  leadership  scores. 
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Regression  Plot 


Actual  By  Predicted  Plot 


Figure  19.  Graph  -  Age/Leadership 


The  P  value  for  the  test  for  significance  of  regression  is  P  =  0.028 
and  RSqr  =  0.58.  This  indicates  evidence  that  age  is  linearly  related  to  leadership. 


Making. 


(3)  Factor:  Situational  Awareness  Predicts  Good  Decision- 


Questions 

•  Decision-Making  Question  (Sessions  2-5/Question  12: 
Items  8  &  9)  -  Participants  were  asked  about  their  opinion 
of  their  teammates  with  regard  to  their  ability  to  identify 
the  enemy’s  center  of  gravity  and  to  attack  it.  Also,  they 
were  asked  about  how  well  the  other  team  members 
adjusted  to  the  changing  enemy  situation.  All  scores  were 
graded  on  a  1-7  Likert  Scale. 

•  Situational  Awareness  Question.  (Sessions  2-5/Question 
12,  Items  16,  17,  &  18)  -  Participants  were  asked  about 
their  confidence  regarding  the  enemy  location,  enemy  size, 
and  direction  of  enemy  movement. 

Results 


Figure  20  clearly  demonstrates  the  strong  correlation  between 
decision-making  and  situational  awareness. 
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Regression  Plot 


Actual  by  Predicted 


Figure  20.  Graph  -  Decision-Making/Situational  Awareness 


The  P  value  for  the  test  for  significance  of  regression  is  P  <  0.0001 
and  RSqr  =  0.97.  This  clearly  indicates  strong  evidence  that  decision-making  is  linearly 
related  to  situational  awareness. 

(4)  Correlation  between  Time  in  Service  (TIS)  and  Leadership. 

Questions 

•  Time  in  Service  Question:  Session  1/Question  15  -  This 
question  asked  for  the  participant  to  state  the  number  of 
years  and  months  he  has  been  in  military  service. 

•  Leadership  Question:  (Sessions  2-5/Question  2). 

Results 

Figure  21  shows  a  strong  correlation  between  TIS  and  leadership. 
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Residual  Plot 


Figure  21. 


Actual  by  Predicted 


Graph  -  Time  in  Service/Leadership 


The  P  value  for  the  test  for  significance  of  regression  is  P  =  0.0226 
and  RSqr  =  0.61.  This  indicates  evidence  that  TIS  (months)  is  linearly  related  to 
leadership. 

e.  TDG/TDS  Summary  of  Results 

Table  1 1  summarizes  the  correlation  results  from  Experiment  2.  P  Values 
below  0.05  indicate  significance. 

Table  11.  Summary  Comparison  of  TDG/TDS  Results  for  Significance 


TDG 

TDS 

P  value 

R- Squared 

P  value 

R-Squared 

Leadership  (Peer  Evaluated) 

VS  Talkativeness  (Peer 
Evaluated) 

0.0005 

0.88 

0.005  0.48 

Strong  Correlation 

Correlation 

Leadership  (Peer  Evaluated) 
VS  Age 

0.0262 

0.59 

0.018 

0.058 

Correlation 

Correlation 

Leadership  (Instructor 
Evaluated)  VS  Leadership 
(Peer  Evaluated) 

0.0143 

0.66 

0.617 

0.04 

Correlation 

No  Correlation 

Situational  Awareness  (Peer 
Evaluated)  VS  Decision 
Making  (Peer  Evaluated) 

0.0004 

0.89 

<0.001 

0.97 

Strong  Correlation 

Strong  Correlation 

Time  in  Service  VS  Leadership 
(Peer  Evaluated) 

0.09 

0.4 

0.0226 

0.61 

Weak  Correlation 

Correlation 
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f.  Qualitative  Analysis 

The  experiment  produced  a  number  of  qualitative  results  that  were 
interesting  and  should  be  included.  These  include  data,  such  as  verbal  participant 
comments  regarding  their  group’s  experience,  researcher  observations,  as  well  as,  the 
participant’s  answers  to  open-ended  questions  from  the  questionnaires.  In  some  cases, 
these  results  were  more  telling  than  the  quantitative  ones. 

(1)  Training  Session  Duration  and  Timing.  The  duration  and 
the  number  of  individual  training  sessions  that  are  required  to  achieve  a  particular  skill 
level  have  been  largely  unexplored.  We  conducted  our  experiment  in  five  total  sessions 
lasting  one  hour  each.  To  begin  the  sessions,  we  read  the  operation  order  and  turned  the 
teams  over  to  3-5  minutes  of  planning  time.  After  their  planning  time  ended,  they  had 
another  3-5  minutes  to  get  their  troops  into  their  starting  positions  before  the  actual 
scenarios  started.  Each  actual  simulation  scenario  lasted  approximately  40  minutes.  We 
knew  that  the  3-5  minutes  of  planning  time  was  not  sufficient  to  create  a  good  tactical 
plan,  but  we  were  constrained  by  the  scenarios  lasting  40  minutes.  The  planning  that  the 
teams  conducted  was  quite  surprising.  We  observed  that  the  TDS  groups  actually  had 
planning  time  left  over  and  were  waiting  to  start  the  actual  scenario.  They  appeared  to  be 
truncating  their  plan  and  rushing  into  the  execution  phase.  This  could  be  attributed  to 
their  heightened  level  of  excitement  when  preparing  to  interact  with  the  simulation.  This 
was  not  only  true  at  the  outset  of  the  CCM  scenarios,  but  during  the  middle  of  the 
sessions,  too.  If  a  group  had  a  plan  to  counter  some  enemy  action,  they  seldom,  if  ever, 
executed  it.  There  seemed  to  be  an  urge  to  forget  about  what  they  had  planned  and  only 
use  CCM  on  a  moment-by-moment  basis. 

(2)  Overall  Session  Time  was  Too  Short.  The  pauses  in  the 
sessions  that  were  required  to  collect  SA  information  ended  up  pushing  the  end  of  the 
simulation  longer  than  expected.  This  left  virtually  no  time  for  participants  to  fill  out 
their  questionnaires.  At  least  10  minutes  should  be  allowed  for  participants  to  properly 
fill  out  the  questionnaires  without  feeling  rushed.  In  some  cases,  the  TDG  group  was 
able  to  complete  their  task  in  the  allotted  1-hour  time  slot,  but  even  they  were  rushed 

during  some  sessions.  The  time  for  each  session  should  be  extended  to  at  least  1.5  hours 
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and  preferably  two  hours.  The  latter  would  allow  the  participants  to  have  sufficient  time 
to  plan,  complete  the  entire  TDG/TDS  scenario,  and  fill  out  their  questionnaires  before 
departing  the  area. 

(3)  Combining  the  Best  of  the  TDG  and  the  TDS.  Military 
schools  that  plan  to  use  this  type  of  training  should  encourage  the  time  tested  and 
superior  planning  process  from  the  TDG  to  take  place  at  the  beginning  of  the  training 
sessions.  Once  unit  leaders  have  established  their  plans,  they  should  implement  them  in 
the  TDS  to  see  how  each  plan  holds  up  under  enemy  contact.  This,  also,  applies  to  how 
the  unit  leaders  and  subordinate  leaders  interact  and  respond  in  the  tactical  virtual 
environment.  The  TDS  will  provide  feedback,  and  an  ultimate  outcome,  which  can  be 
discussed  in  an  after  action  review  in  terms  of  the  original  plan.  Subject  matter  experts 
should  be  present  throughout  to  prepare  for  a  robust  after  action  review  session. 

(4)  Team  Cohesiveness.  We  noticed  improvement  in  team 
cohesiveness  over  the  three  days  and  five  separate  sessions.  During  the  first  session,  the 
TDS  sub-groups  were  somewhat  hesitant  to  communicate  with  one  another  before  and 
during  the  battle.  Each  participant  seemed  to  be  focused  on  his  particular  task  and  paid 
little  attention  to  his  teammates.  After  the  final  session,  there  was  a  good  deal  of  team 
cohesion  among  all  groups.  Particularly  in  the  final  TDS  session,  TDS  Group  1  was 
victorious  and  burst  out  of  their  seats  to  slap  hands  and  congratulate  each  other. 

(5)  Confusion  and  Loss  of  Situational  Awareness.  Some  TDS 
participants  had  trouble  keeping  track  of  which  units  were  friendly  and  which  were 
enemy.  In  their  view,  they  did  not  know  which  squads  or  units  were  on  their  side.  At 
times,  their  confusion  led  them  to  fire  on  their  own  adjacent  units.  This  was  obviously 
not  an  issue  in  the  TDG  group.  This  could  be  used  as  an  important  teaching  point 
regarding  the  confusion  of  combat  and  how  this  could  contribute  to  friendly  fire 
incidents.  Instructors  could  emphasize  how  important  it  is  to  maintain  SA  at  all  times. 

(6)  Pleasant  Training  Experiences  Contribute  to  Learning.  Not 
all  military  training  can  be  fun.  To  emphasize  that  there  are  times  when  the  mission  has 
priority  even  at  the  expense  of  one’s  comfort,  some  military  training  must  be  unpleasant. 
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However,  some  training  industry  experts  believe  that  a  pleasurable  training  experience 
can  contribute  to  participant  learning.  According  to  Michael  Allen  of  Allen  Interactions, 
a  Minneapolis  training  firm,  “People  remember  things  through  humor.”  “When  you  talk 
about  a  lasting  learning  experience,  you  recall  what  is  notable  about  the  event.  If 
something  unusual  or  funny  happens,  you'll  probably  remember  it”  (Fister,  1999). 

During  the  final  after  action  review  and  once  all  sessions  had 
concluded,  the  TDS  participants  reported  they  were  motivated  and  enjoyed  the 
experience.  Conversely,  the  TDG  group  reported  they  became  bored  with  it.  This 
boredom  was  a  fact  supported  by  some  of  the  note  pages  we  collected  from  the  TDG 
group  after  their  sessions.  Some  had  extensive  doodle  marks  on  them  indicating  that  this 
particular  person  was  able  to  “wander  off’  mentally  from  the  planning  task  in  which  he 
was  involved.  In  part  of  our  final  questionnaire,  we  asked  participants  in  both  TDG  and 
TDS  groups  about  the  quality  of  their  individual  training  experiences.  The  scores  in 
Table  12  were  taken  from  the  final  questionnaires  and  illustrate  interesting  differences 
between  the  two  approaches. 

Table  12.  Comparison  of  Quality  of  Training 


Question 

Average  Score 
(1-7  Likert) 

Question 

Average  Score 
(1-7  Likert) 

"When  you  think  about  your  future 
training  needs,  to  what  extent  would  you 
like  it  to  be  supplemented  with  TDGs?” 

2.75 

“When  you  think  about  your  future 
training  needs,  to  what  extent  would  you 
like  it  to  be  supplemented  with  computer 
based  simulations  like  CCM?” 

4.625 

“When  you  think  about  your  future 
training  needs,  to  what  extent  would  you 
like  it  to  be  supplemented  with  computer 
based  simulation  of  a  TDG?” 

5 

*  Question  Not  Applicable  * 

"How  well  did  the  TDG  sessions  meet 
your  expectations  regarding  training 
value?" 

3.1 

"How  well  did  the  TDS  sessions  meet 
your  expectations  regarding  training 
value?" 

4.5 

"The  TDG  experience  was  able  to 
provide  me  with  a  good  mental  picture  of 
the  battle  space." 

3.3 

"Rate  the  simulation's  ability  to  provide 
you  with  VISUAL  cues  so  that  you  can 
identify  all  threats  within  the  mission." 

4.9 

"Rate  the  simulation  ability  to  provide 
you  with  AUDIO  cues  so  that  you  can 
identify  all  threats  within  the  mission." 

4.6 

"In  your  opinion,  what  was  the  overall 
quality  of  training  in  the  TDG?" 

3.2 

"In  your  opinion,  what  was  the  overall 
quality  of  training  in  the  TDS?" 

4.9 
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Table  12  indicated  a  desire  by  all  participants  to  incorporate 
computer  based  simulation  into  their  training.  What  makes  this  so  significant  is  that  the 
TDG  group  indicated  displeasure  with  their  task  and  opted  for  an  alternative  to  it  in 
computer-based  simulation.  The  TDS  participants  enjoyed  their  training  enough  to 
continue  to  recommend  it  be  included  in  future  tactical  decision-making  training 
regimens.  The  scores  in  Table  12,  also,  indicate  that  the  perception  of  the  quality  of 
training  was  lower  in  the  TDG  group  than  in  the  TDS  group. 

(7)  Verbal  and  Non-Verbal  Communications.  Some  of  the 
participants’  communications  (verbal  and  non-verbal),  and  interactions  during  the 
sessions,  warrant  comment.  Our  intent  was  to  simulate  the  communications  between 
platoon  leaders  and  squad  leaders  in  the  most  realistic  way  possible.  In  a  field 
environment,  a  platoon  leader  usually  traveled  with  one  of  his  squads.  This  prevented 
him  from  seeing  exactly  what  the  other  two  squad  leaders  were  observing  at  any 
particular  point  in  time.  We  simulated  this  by  advising  participants  before  the  sessions 
began  that  each  squad  leader  should  remain  at  his  own  computer  and  only  look  at  his  own 
screen.  Likewise,  the  platoon  leaders  should  remain  with  a  single  squad  throughout  and 
not  hop  back  and  forth  between  screens.  Verbal  communication  would  be  heard  by  all  as 
if  they  were  all  on  the  same  radio  network.  This  did  not  occur.  During  the  sessions,  we 
constantly  reminded  the  squad  leaders  to  remain  in  their  seats  and  to  keep  their  focus  on 
their  screens  (their  view  of  the  battle  space).  However,  they  got  up  to  help  their  fellow 
squad  leaders  who  were  having  trouble  or  who  were  unable  to  understand  what  was  going 
on  in  their  view  of  the  simulation.  In  like  manner,  the  platoon  leaders  were  consistently 
looking  over  at  their  other  squad  leader’s  computer  screens  in,  what  we  believe,  was  an 
attempt  to  gamer  more  SA  before  they  made  decisions. 

5.  Summary 

The  following  five  points  were  significant  findings  from  the  summary  of 
quantitative  and  qualitative  results  in  Experiment  2. 

•  In  the  longitudinal  analysis  across  all  sessions,  we  observed  that 
participants  received  their  highest  leadership  scores  when  they  were 
designated  as  the  leader  in  the  TDG  and  the  TDS  sessions.  However,  the 
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TDS  participants  had  a  much  higher  deviation  from  the  mean  on 
leadership  because  they  were  acting  in  the  role  of  a  leader  by  standing, 
issuing  orders,  and  supervising  their  squad  leaders  during  the  virtual 
battle. 

•  TDS  participants  seemed  to  equate  a  favorable  outcome  in  the  simulation 
with  good  decision-making.  This  may  or  may  not  be  true,  but  their  scores 
consistently  supported  this  concept.  We  found  that  peer  evaluations  were 
strongly  correlated  between  the  participants’  decision-making  and 
situational  awareness  scores. 

•  We  tested  the  hypothesis  that  there  was  a  difference  between  the  TDG  and 
TDS  leadership,  decision-making,  and  SA  scores.  We  believed  that  the 
methods  were  the  cause  of  differences  with  respect  to  decision-making 
and  situational  awareness  when  evaluated  by  peers.  When  we  evaluated 
the  groups,  there  was  no  significant  difference  between  the  methods  when 
examining  decision-making  and  leadership.  We  were  not  able  to  evaluate 
if  there  was  a  significant  difference  between  the  groups  on  situational 
awareness  because  we  were  unable  to  obtain  SA  data  from  the  TDG 
group. 

•  Internal  to  the  TDG  and  TDS  groups,  we  found  that  perceived  leadership 
was  correlated  with  talkativeness,  time-in-service,  and  age.  SA  and 
decision-making  were  strongly  correlated  in  both  groups. 

•  Our  qualitative  assessment  determined  that  an  improved  approach  to 
training  with  tactical  decision-making  simulations  should  be  considered. 
We  believe  that  a  new  approach,  which  we  call  the  Integrated  Tactics  and 
Planning  Simulation  Exercise  (ITAPSE),  would  bridge  the  best  of  both 
training  approaches  (TDS  &  TDG).  The  TDS  was  conducted  almost 
exclusively  within  the  virtual  environment  and  seemed  to  be  more 
engaging  to  the  participants.  The  TDS  experience  was  exciting  and 
allowed  each  participant  to  play  a  highly  active  role.  The  TDG  was 
conducted  with  pen  and  paper.  It  prompted  more  thought  about  individual 
actions  and  fostered  more  thorough  planning.  The  ITAPSE  combines  the 
planning  techniques  of  the  TDG  with  the  ability  to  implement  and  execute 
that  plan  via  the  TDS. 
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IV.  RECOMMENDATIONS  AND  FOLLOW-UP  EXPERIMENT 


A.  RECOMMENDATIONS 

After  the  experiment  and  data  analysis  were  completed,  there  were  several 
recommendations  that  we  identified.  These  fall  into  the  following  categories: 

1.  Improvements  to  the  CCM  Software  for  TDS  Use 

We  believe  that  if  CCM  is  to  continue  to  be  used  as  a  TDS  for  training,  the 
following  modifications  to  the  code  and  meta-data  would  provide  for  a  richer  experience 
for  the  students: 


a.  Operation  Orders 

Administrators  should  ensure  that  the  operation  orders  are  corrected  so 
that  the  information  is  accurate  and  consistent  for  both  groups.  The  orders  used  in  this 
experiment  contained  numerous  acronyms  and  jargon  that  was  specific  to  an  infantry 
environment.  This  would  be  acceptable  for  military  personnel  who  have  been  trained  in 
infantry  TTPs,  but  not  for  members  of  other  services  and  especially  not  for  Foreign 
Service  members. 

Some  information  that  would  be  critical  to  the  TDG  group  was  excluded 
entirely  from  the  orders.  The  orders  were  adapted  for  use  by  the  TDG  group,  but  were 
originally  written  by  the  CCM  developers  for  the  TDS  group.  Information,  such  as  the 
current  location  of  one’s  own  unit,  was  not  included.  The  TDS  group  would  immediately 
understand  where  their  unit  was  located  when  they  started  CCM,  but  the  TDG  group  had 
no  ability  to  get  this  information.  We  had  to  estimate  the  TDG  groups’  locations  during 
the  TDG  sessions  to  give  them  a  starting  point. 

Orders,  also,  had  conflicting  and  erroneous  information.  The  operation 
orders  that  we  used  are  located  in  Appendix  K.  Session  1-la’s  Operation  Order  stated 
that  the  company  had  been  provided  no  mortar  support,  yet  the  Task  Section  of  the  order 
stated  plainly  that  mortar  priority  of  fire  was  to  third  platoon. 
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b.  Pause  Feature 

When  the  current  system  was  paused,  it  was  difficult  to  establish  ground 
truth  of  unit  locations  because  each  participant’s  screen  was  on  a  different  part  of  the 
battlefield.  Internal  simulation  algorithms,  that  display  only  what  the  participant’s  team 
avatars  within  the  simulation  are  capable  of  seeing  based  on  distances  and  terrain 
features,  controlled  the  number  of  troops  visible  to  the  individual. 

c.  Integrated  Situational  Awareness  Measurement  Tool  (ISAM-T) 

Future  editions  of  CCM  should  incorporate  a  built-in  SA  evaluation  tool 
that  will  present  an  SA  screen  to  the  user  at  preset  or  instructor  controlled  points  during 
the  sessions.  The  GRADE  SA  evaluation  tool  used  in  this  experiment  was  difficult  to 
administer  because  the  TDS  did  not  have  a  good  mechanism  to  pause  the  simulation.  On 
two  separate  occasions,  users  accidentally  quit  the  TDS  and  forced  a  complete  restart  of 
the  scenario.  This  wasted  approximately  20  minutes  of  session  time. 

Additionally,  ISAMT  could  internally  calculate  distances,  size,  and 
locations  of  friendly/enemy  units  and  compare  this  to  ground  truth.  This  will  return  an 
objective  difference  score  between  where  the  student  thought  the  enemy/friendly  forces 
were  and  where  they  actually  were.  Based  on  this  criteria,  the  simulation  will  grade  the 
student  and  provide  an  objective  measure  of  SA.  This  information  could,  also,  be  saved 
and  used  by  instructors/SMEs  in  the  AAR  portion  of  the  scenario. 

d.  XML  Output  File 

“Many  simulations  provide  none  or  only  rudimentary  support  for  AAR” 
(Sadagic,  2007).  CCM  does  provide  some  after  action  information  output  to  the  screen  at 
the  end  of  the  simulation,  but  here  is  no  file  on  which  a  researcher  could  run  statistics  to 
look  for  possible  trends.  We  suggest  incorporating  an  XML  output  file  that  would 
provide  results  for  each  player  in  CCM.  This  could  be  used  to  extract  objective  data  for 
each  player  in  the  team.  It  could  provide  a  way  to  evaluate  their  tactical  decision-making 
in  terms  of  distances  covered  at  a  run,  number  of  rounds  expended  per  unit  time,  unit 
proximity  to  adjacent  units,  and  whether  or  not  the  student  fired  on  friendly  troops. 
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e.  Instructor/Administrative  Workstation 

CCM  should  include  an  administrative  workstation  that  would  allow  for 
instructor’s  observation  and  control  of  friendly  and  enemy  units.  This  would  allow  the 
instructor  to  maintain  oversight  of  both  sides  during  the  entire  scenario.  From  this 
station,  he  would  be  able  to  pause  the  TDS  and  administer  the  automated  SA  evaluation 
that  we  suggested  as  a  new  feature. 

f.  True  Record/Playback 

We  recommend  incorporating  a  true  record/playback  feature  for  action 
review  purposes  into  this  master  workstation.  The  current  playback  feature  within  CCM 
has  some  level  of  interaction  available  to  it  (the  user  can  interact  with  the  recorded 
session  as  it  is  playing  and  alter  the  outcome).  Additionally,  in  the  current  CCM  record 
feature,  time  is  constant  and  non-variable  (1  second  =  1  second).  We  recommend  that 
developers  include  a  fast-forward  capability  so  that  TDS  administrators  can  move  quickly 
to  critical  points  in  the  battle  to  illustrate  specific  learning. 

g.  Virtual  White  Board 

We  recommend  including  a  virtual  white  board  overlay.  Instructors  would 
draw  on  the  master  screen  to  illustrate  tactical  accomplishments  or  mistakes.  The 
instructor  screen  drawing  could  show  up  on  some  or  all  students’  screens. 

h.  Re-designation  of  Unit  Leadership 

If  the  actual  platoon  leader  is  killed  in  battle,  CCM  should  allow  for  the 
designation  of  another  platoon  leader.  CCM  has  leadership  rings  both  around  each  team 
and  team  leader  to  indicate  that  certain  individuals  have  more  leadership  capability  than 
others.  The  proximity  of  troops  to  their  leader  improves  their  morale.  If  a  team  leader 
dies  in  battle,  his  troops  will  be  more  likely  to  panic  in  the  face  of  enemy  fire. 
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2.  Potential  Bugs  in  CCM. 
a.  Observer  Mode 

We  believe  we  identified  a  bug  in  the  software.  During  one  of  the 
sessions,  two  players'  screens  went  into  observer  mode.  This  term  “observer  mode” 
describes  a  condition  where  visualization  on  screen  was  present,  but  interaction  with  the 
system  was  not  possible  for  the  participant.  One  reason  for  this  resulted  when  a 
participant’s  entire  squad  was  killed  by  enemy  fire  and,  therefore,  had  no  troops  to 
control.  However,  the  other  reason  for  the  case  of  “observer  mode”  was  undetermined. 
We  looked  through  the  manuals  for  indications  as  to  why  this  would  have  happened,  but 
found  none. 


b.  Potential  Memory  Leak  in  CCM 

We  noticed  some  anomalies  with  the  networked  configuration  of  CCM 
during  the  TDS  sessions.  Future  administrators  of  the  experiment  should  ensure  that 
computers  are  freshly  booted  before  starting  each  session.  On  one  occasion,  to  reduce  the 
amount  of  setup  time,  we  left  the  machines  on  overnight  and  at  the  network  setup  screen. 
After  this  long  period  of  computer  inactivity,  some  of  the  computers  would  not 
synchronize  with  the  host  in  multiplayer  mode.  We  finally  had  to  reboot  all  six 
computers  before  CCM  would  synchronize  over  the  network. 

B.  SUGGESTIONS  FOR  A  FOLLOW-UP  EXPERIMENT 

If  future  research  efforts  try  to  replicate  this  study,  or  to  examine  certain 
components  in  more  detail,  then  researchers  might  want  to  consider  the  following: 

1.  Participants 

Search  for  a  pool  of  subjects  that  are  closer  to  the  actual  population  for  which 
TDS  training  will  be  implemented.  This  might  be  used  to  confirm  the  previous  findings 
in  this  study.  The  follow-on  study  should  be  conducted  at  a  site  where  a  traditional 
training  approach  to  tactical  decision-making  is  already  being  used  so  that  the  training 
approaches  can  be  easily  compared  and  measured. 
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2.  Use  of  Subject  Matter  Experts  (SMEs) 

It  would  be  good  to  have  two  SMEs  per  TDS  subgroup  who  will  comment  on  and 
guide  discussions  during  the  simulation.  Even  more  importantly,  they  can  conduct  after 
action  review  (AAR)  sessions  and  speak  from  their  experiences.  The  CCM  TDS  was 
developed  with  doctrinally  sound  infantry  TTPs,  but  feedback  from  the  system  in  this 
regard  was  not  presented  to  the  users  during  our  experiment.  Infantry  SMEs  are  an 
essential  part  of  training  when  using  a  TDS. 

3.  Integrated  Tactics  and  Planning  Simulation  Exercise  (ITAPSE) 

We  strongly  suggest  combining  the  best  features  of  both  the  TDS  and  the  TDG 
into  TAPSE.  This  approach  would  incorporate  the  superior  methodical  planning 
techniques  of  the  TDG  with  the  ability  to  implement  one’s  plan  and  receive  feedback 
provided  by  the  TDS.  Dr.  Sadagic  (2007)  of  the  Naval  Postgraduate  School,  MOVES 
Institute  states,  “...practitioners  should  be  encouraged  and  offered  time  to  explore  a 
synergy  of  old  (proven)  training  environments  they  know  very  well  and  simulations.” 
This  quote,  and  our  findings  in  Experiment  2,  underscores  the  need  to  build  a  combined 
approach  to  tactical  decision-making  training. 

4.  Focus  on  Planning 

We  recommend  that  planning  be  examined  in  two  different  ways:  between  a  TDG 
and  TDS  group  and  between  a  traditional  TDS  group  and  one  that  implemented  ITAPSE. 

Another  suggestion  would  be  to  vary  the  planning  time  for  each  group  and,  also, 
to  look  for  correlations  between  which  participants  quickly  gave  up  their  primary  plan 
compared  to  the  time  they  spent  creating  their  plan.  Record  the  reasons  participants 
provide  for  their  decision  to  change  their  plans. 

Future  researchers  might  consider  evaluating  how  quickly  participants  change 
their  plans  if  they  did  not  seem  to  be  working.  Then,  record  the  rationale  that  led  them  to 
believe  that  their  plans  were  not  working. 

Finally,  they  might  evaluate  whether  participants  tended  to  stick  with  their 
original  plan  or  if  there  was  a  tendency  to  disregard  it  at  the  earliest  opportunity.  Record 
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the  time  it  took  participants  to  realize  that  their  original  plan  was  not  working  and  that  the 
situation  required  a  shift  to  an  alternative  COA.  If  they  chose  to  deviate  from  their  plan, 
did  they  have  a  justifiable  reason  to  do  so? 

5.  Examine  Personality  Factors 

TDS  participants  had  trouble  keeping  track  of  which  units  were  friendly  and 
which  were  enemy.  Future  studies  could  consider  evaluating  personality  traits  to 
determine  whether  there  is  a  correlation  with  a  participant’s  tendency  to  rush  into  battle 
or  to  dig-in  and  wait. 

6.  TDS  After  Action  Review  (AAR) 

Incorporate  an  after-action  review  session  into  each  TDS  session.  A  sufficient 
amount  of  time  should  be  included  in  the  sessions  in  order  to  accommodate  this 
discussion. 

We  believe  that  a  thorough  examination  of  the  previous  items,  within  the  context 
of  a  follow-on  study,  could  yield  many  interesting  results  such  as  a  correlation  between 
extroversion  and  perception  of  leadership  performance  and/or  a  successful  outcome 
within  the  simulation.  Additionally,  future  research  might  determine  that  a  lengthy 
planning  session  correlates  to  more  resistance  to  changing  the  plan  upon  enemy  contact. 
Finally,  the  ITAPSE  method  of  training  decision-making  skills  could  offer  students  the 
best  that  TDG  and  TDS  to  offer. 
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APPENDIX  A  -  CONSENT  FORM  (EXPERIMENT  1) 


Naval  Postgraduate  School 

Participant  Consent  Form  & 

Minimal  Risk  Statement 

Introduction.  You  are  invited  to  participate  in  a  study  entitled  Evaluation  of  the  Preliminary  Training  for 
Virtual  Simulations  being  conducted  by  the  Naval  Postgraduate  School  MOVES  Institute. 

Procedures.  We  will  use  the  Human  Systems  Integration  Laboratory  (HISL)  to  conduct  an  experiment 
examining  three  different  training  methods.  We  will  evaluate  training  approaches  for  operating  a  Tactical 
Decision-Making  Simulation  (TDS).  The  TDS  is  called  Close  Combat  Marine  (CCM)  version  4.0  and  is  a 
virtual  environment  (a  simulation)  used  by  the  USMC  to  train  Marines  in  tactical  decision-making.  The 
researchers  will  provide  the  participants  with  a  brief  (no  more  than  5  minutes)  explanation  of  the  TDS  and 
what  they  will  be  doing  during  the  simulation.  We  will  collect  user  computer  simulation  experience  and 
demographics  in  the  form  of  a  questionnaire  prior  to  the  commencement  of  hase  I. 

Phase  I  will  consist  of  a  learning  scenario  where  participants  will  learn  how  to  operate  the  TDS.  This  phase 
will  last  approximately  30  minutes. 

In  Phase  II,  the  participants  will  go  through  a  scenario  a  second  time. 

In  the  final  phase,  Phase  III,  the  researchers  will  administer  a  participant  survey.  Each  participant  will  go 
through  only  one  of  the  scenarios  for  Phase  I  and  one  scenario  for  phase  II,  a  total  of  2  scenarios. 

The  researchers  will  record  participants’  actions  using  a  video  camera  (Note:  participants  themselves  will 
not  appear  in  the  video). 

Risks  and  Benefits.  I  understand  that  this  project  does  not  involve  greater  than  minimal  risk  and  involves 
no  known  reasonably  foreseeable  risks  or  hazards  greater  than  those  encountered  in  everyday  life.  I  have, 
also,  been  informed  of  any  benefits  to  myself  or  to  others  that  may  reasonably  be  expected  as  a  result  of 
this  research. 

Compensation.  I  understand  that  no  tangible  reward  will  be  given.  I  understand  that  a  copy  of  the 
research  results  will  be  available  at  the  conclusion  of  the  experiment. 

Confidentiality  &  Privacy  Act.  I  understand  that  all  records  of  this  study  will  be  kept  confidential  and 
that  my  privacy  will  be  safeguarded.  No  information  will  be  publicly  accessible  which  could  identify  me 
as  a  participant  and  I  will  be  identified  only  as  a  code  number  on  all  research  forms.  I  understand  that 
records  of  my  participation  will  be  maintained  by  NPS  for  5  years,  after  which  they  will  be  destroyed. 

Voluntary  Nature  of  the  Study.  I  understand  that  my  participation  is  strictly  voluntary  and,  if  I  agree  to 
participate,  I  am  free  to  withdraw  at  any  time  without  prejudice. 

Points  of  Contact.  I  understand  that  if  I  have  any  questions  or  comments  regarding  this  project  upon  the 
completion  of  my  participation,  I  should  contact  the  Principal  Investigator,  C.  Neil  Fitzpatrick, 
Capt/USMC,  (831)  917-6663,  cnfitzpa@nps.edu,  or  Umit  Ayvaz,  Capt/Turkish  Army,  uayvaz@nps.edu. 
Any  medical  questions  should  be  addressed  to  LTC  Eric  Morgan,  MC,  USA,  (CO,  POM  Medical  Clinic), 
(831)  242-7550,  eric.morgan@nw.amedd.army.mil. 
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Statement  of  Consent.  I  have  read  and  understand  the  above  information.  I  have  asked  all  questions  and 
have  had  my  questions  answered.  I  agree  to  participate  in  this  study.  I  will  be  provided  with  a  copy  of  this 
form  for  my  records. 

Participant’s  Name  (please  print) 

Participant’s  Signature  Date 

Researcher’s  Signature  Date 
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APPENDIX  B  -  IRB  REQUEST  (EXPERIMENT  1) 


C.  Neil  Fitzpatrick,  Capt/USMC 
Umit  Ayvaz,  Capt/Turkish  Army 
MOVES 
Watkins  Hall 

Naval  Postgraduate  School 
Monterey,  California  93943 


cnfitzpa@nps.edu 

(831)  917-6663 

uay  vaz@nps .  edu 

(831)-393-1272 


To:  Protection  of  Human  Subjects  Committee 

Subject:  Application  for  Human  Subjects  Review  (Title):  The  most  appropriate  time  and 
method  for  becoming  comfortable  with  the  Tactical  Decision-Making  Simulation  (TDS) 
controls. 

1 .  Attached  is  a  set  of  documents  outlining  a  proposed  experiment  to  be  conducted 
by  C.  Neil  Fitzpatrick  (Captain  USMC  )  and  Umit  AYVAZ  (Captain  Turkish  Army) 
for  their  final  project  in  OA3402  Research  Methods  in  Performance  Assessment  class 
in  HSIL  lab  in  Glasgow  Hall. 

2.  We  are  requesting  approval  of  the  described  experimental  protocol.  For  your 
reference,  an  experimental  outline  is  included  that  describes  the  methods  and 
measures  we  plan  to  use. 

3.  We  include  the  consent  forms,  privacy  act  statements,  all  materials  and  forms 
that  a  subject  will  read  or  fill-out,  and  the  debriefing  forms  (if  applicable)  we  will  be 
using  in  the  experiment. 

4.  We  understand  that  any  modifications  to  the  protocol  or  instruments/measures 
will  require  submission  of  updated  IRB  paperwork  and  possible  re-review.  Similarly, 
we  understand  that  any  untoward  event  or  injury  that  involves  a  research  participant 
will  be  reported  immediately  to  the  IRB  Chair  and  NPS  Dean  of  Research. 


Dr.  Amela  Sadagic,  Principal  Investigator  (PI) 
C.  Neil  Fitzpatrick,  Student  Researcher 
Umit  AYVAZ,  Student  Researcher 
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APPLICATION  FOR 

HUMAN  SUBJECTS  REVIEW  (HSR) 


HSR  NUMBER  (to  be  assigned) 


PRINCIPAL  INVESTIGATOR(S)  (Full  Name,  Code,  Telephone) 

Dr.  Amela  Sadagic,  MOVES  Research  Associate  Professor,  831.656.3819 


APPROVAL  REQUESTED 


[  X  ]  New 


[  ]  Renewal 


LEVEL  OF  RISK  [  ]  Exempt  [  X  ]  Minimal  [  ]  More  than  Minimal 
Justification: 


WORK  WILL  BE  DONE  IN  GLASCOW 
Human  Systems  Integration  Lab. 


ESTIMATED  NUMBER  OF  DAYS  TO 
COMPLETE 

5  days _ 


MAXIMUM  NUMBER  OF  SUBJECTS 
15 


ESTIMATED  LENGTH  OF 
SUBJECT’S  PARTICIPATION 
300  minutes 


EACH 


SPECIAL  POPULATIONS  THAT  WILL  BE  USED  AS  SUBJECTS 

[  ]  Subordinates  [  ]  Minors  [  X  ]  NPS  Students  [  ]  Special  Needs  (e.g.  Pregnant 
women) 

Specify  safeguards  to  avoid  undue  influence  and  protect  subject’s  rights: 


OUTSIDE  COOPERATING  INVESTIGATORS  AND  AGENCIES 
[  ]  A  copy  of  the  cooperating  institution’s  HSR  decision  is  attached. 


TITLE  OF  EXPERIMENT  AND  DESCRIPTION  OF  RESEARCH  (attach  additional  sheet 
if  needed). 

Introduction  to  Tactical  Decision-Making  Simulations  (TPS):  Pointing  and  Clicking 

INTRODUCTION 

We  will  use  the  Human  Systems  Integration  Laboratory  (HISL)  to  conduct  an  experiment 
examining  3  different  training  methods.  We  will  evaluate  training  approaches  for  operating  a 
Tactical  Decision-Making  Simulation  (TDS).  The  TDS  is  called  Close  Combat  Marine 
(CCM)  version  4.0  and  is  a  virtual  environment  (a  simulation)  used  by  the  USMC  to  train 
Marines  in  tactical  decision-making.  Researchers  will  provide  the  participants  with  a  brief 
(no  more  than  5  minutes)  explanation  of  the  TDS  and  how  they  will  participate  in  the 
simulation. 

We  will  administer  and  collect  user  computer  simulation  experience  and  demographics  in  the 
form  of  a  Pre-TDS  questionnaire  prior  to  the  commencement  of  Phase  I. 

The  overall  study  will  be  divided  into  3  phases: _ 
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PHASE  I 

Participants  will  engage  in  one  of  the  3  TDS  training  methodologies  to  learn  how  to  operate 
the  TDS.  Phase  I  scenarios  will  include: 

a)  TDS  Boot  Camp  scenario  alone  (provided  by  the  TDS  manufacturer), 

b)  TDS  Boot  Camp  scenario  with  memory-sheet, 

c)  Memory-sheet  alone  with  a  practice  scenario  (user  progresses  at  his  own  pace  and 
asks  questions  of  researchers  if  he  gets  stuck). 

This  phase  will  last  approximately  30  minutes. 

PHASE  II 

All  participants,  regardless  of  Phase  I  grouping,  will  all  go  through  the  same  phase  II 
scenarios  to  provide  a  baseline  for  comparison  of  the  different  groups. 

PHASE  III 

Phase  III  is  the  final  phase  of  the  study.  The  researchers  will  administer  a  Post-TDS 
participant  survey.  Each  participant  will  go  through  only  one  of  the  scenarios  for  Phase  I  and 
one  scenario  for  Phase  II  for  a  total  of  2  scenarios. 

The  researchers  will  record  participants’  actions  using  a  video  camera  (Note:  participants 
themselves  will  not  appear  in  the  video.)  The  recording  of  sessions  will  be  both  audio  and 
video.  Again,  the  user’s  image  will  not  appear  in  the  video.  The  video  will  record  the 
computer  screen  that  the  participant/user  is  interacting.  This  video  will  be  used  afterwards  to 
evaluate  the  efficacy  of  the  training  method.  The  number  of  mistakes  will  be  one  of  the 
objective  metrics  used  to  determine  how  well  the  participants  learned  the  task. 

The  researchers  will,  also,  subjectively  evaluate  each  participant  on  his/her  performance 
using  a  Likert  scale. 


I  have  read  and  understand  NPS  Notice  on  the  Protection  of  Human  Subjects.  If  there  are  any 
changes  in  any  of  the  above  information  or  any  changes  to  the  attached  Protocol,  Consent 
Form,  or  Debriefing  Statement,  I  will  suspend  the  experiment  until  I  obtain  new  Committee 
approval. 

SIGNATURE _ ,  PI 

SIGNATURE _ ,  Student  Researcher 

SIGNATURE _ ,  Student  Researcher 

DATE 
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APPENDIX  C  -  SURVEY  (EXPERIMENT  1) 


Pre-TDS  Survey 

Please  fill  in  the  following  questionnaire.  Answer  all  questions  as  objectively  as  you  can.  All 
information  will  be  held  confidential. 

If  you  need  to  expand  any  answer  please  use  the  reverse  side  of  your  paper  sheet. 


SIMULATION  EXPERIENCE 


1.  Were  you  required  to  use  computer-based  training  simulations  as  part  of  your  training  in  the  past? 
YES  /  NO  (circle  one) 
a.  If  YES: 


1)  What  type  simulations  have  you  used  and  what  skills  were  they  used  to  train? 


1.  Tactical  level 

2.  Operational  level 

3.  Strategic  level 

Describe  it: 

Describe  it: 

Describe  it: 

2)  Tactical:  What  was  the  total  number  of  hours  you  were  expected  to  use  them  (estimate)? 
_ hours 

Operational:  What  was  the  total  number  of  hours  you  were  expected  to  use  them  (estimate)?  _ 

hours 

Strategic:  What  was  the  total  number  of  hours  you  were  expected  to  use  them  (estimate)?  _ hours 

3)  That  number  of  hours  was  (circle  one  number  on  the  scale  from  1  to  5)  to  acquire  a  good  skill  level: 


1  =  not  at  all 

2  =  less  than 

3  =  sufficient 

4  =  more  than 

5  =  Totally 
sufficient 

sufficient 

sufficient 

sufficient 

4)  Add  any  comment  you’d  like  to  justify  that  answer: 


2.  Do  you  have  experience  with  playing  commercial  video  games?  YES  /  NO  (circle  one) 

3.  If  YES, 
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a)  What  kind  of  video  games  do  you  play?  (circle  all  that  apply) 


1.  first 

2.  flight 

3. 

4. 

5.  puzzle, 

6. 

7.  online 

8. 

9. 

person 

shooter 

simulations 

racing 

other 

sports 

strategy, 
card,  board 

adventure, 
fantasy,  role 
playing 

multiplayer 

games 

arcades 

other 

b)  How  often  do  you  play  them? 

Circle  one:  several  times  a  week  /  once  a  week  /  several  times  a  month  /  once  a  month  /  several  times  a 
year 

c)  That  is  about _ (number)  hours  per:  day  /  week  /  month  /  year  (circle  one). 

4.  Do  you  have  your  own  personal  computer  or  access  to  a  computer  that  you  can  use  on  a  daily  basis?  YES 
/  NO  (circle  one) 

a.  If  YES,  how  long  have  you  been  using  it  on  daily  basis?  For _ (number)  years  and 

_ (number)  months. 


5.  What  hand  do  you  use  to  operate  a  computer  mouse?  LEFT  /  RIGHT  /  I  am  good  with  either  (circle 
one) 


6.  How  often  do  you  use  email,  browse  the  web,  or  use  some  other  computer  applications?  (circle  the  answer  in 
bold  that  applies  to  you) 


Knowledge  and  skills:  .  Your  current  level 


Monthly  Daily 

(Infrequent)  (Very  Frequent) 


A. 

Email 

after  school  hours 
(private  use) 

1 

2 

3 

4 

5 

at  school  (in  the  unit) 

1 

2 

3 

4 

5 

B. 

Browse  the  web 

after  school  hours 
(private  use) 

1 

2 

3 

4 

5 

at  school  (in  the  unit) 

1 

2 

3 

4 

5 

c. 

Office  automation  (Word,  Excel, 
PPT) 

after  school  hours 
(private  use) 

1 

2 

3 

4 

5 

_ 

at  school  (in  the  unit) 

1 

2 

3 

4 

5 

D. 

(some  other  application  -  which?) 

after  school  hours 
(private  use) 

1 

2 

3 

4 

5 

at  school  (in  the  unit) 

1 

2 

3 

4 

5 

E. 

(some  other  application  -  which?) 

after  school  hours 
(private  use) 

1 

2 

3 

4 

5 

_ 

at  school  (in  the  unit) 

1 

2 

3 

4 

5 
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7.  When  you  think  about  your  own  knowledge  of  the  procedures,  and  your  current  skills,  how  would  you  rank 
them? 


# 

Knowledge  and  skills: 

.  Your  current  level 

worst 

best 

1. 

Familiarity  with  a  five  paragraph  order 

1 

2 

3 

4 

5 

2. 

Employ  Machine  guns  in  support  of 
Offensive  Operations 

1 

2 

3 

4 

5 

3. 

Employ  Machine  guns  in  support  of 
Defensive  Operations 

1 

2 

3 

4 

5 

4. 

Conduct  a  Infantry  Squad/Platoon  Attack 

1 

2 

3 

4 

5 

5. 

Control  tactical  movement  of  a 

Squad/Platoon  in  the  Offense 

1 

2 

3 

4 

5 

6. 

Conduct  a  Squad/Platoon  Attack  on  Urban 
Terrain 

1 

2 

3 

4 

5 

7. 

Conduct  a  Squad/Platoon  Defense  on  Urban 
Terrain 

1 

2 

3 

4 

5 

DEMOGRAPHICS 


1.  Subject#  =  (You  will  be  provided  with  a  uniquely  coded  number) : _ 

2.  Date: _  Time: _ 

3.  Year  of  birth:  _ 

4.  Country  of  Citizenship: _ 

5.  Service  component:  (circle  one)  Army  Navy  Air  Force  Marine  Corps  Coast 

Guard  Civilian 

6.  Briefly  describe  your  Primary  MOS,  (Example:  Infantry,  Surface  Warfare,  communications,  etc): 


7.  NPS  Curriculum: _ 

8.  Your  Rank:  _ 

9.  Total  number  of  months  in  service:  _ 

Thank  you  very  much  for  participating  in  the  study.  Relax  and  Enjoy! 
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Post-TDS  Survey 


1.  Date: _  Time: _ 

2.  Subject#  =  (You  will  be  provided  with  a  uniquely  coded  number) : _ 

3.  What  training  methodology  was  administered  during  your  session:  (please  check  only  one  block) 


4.  That  amount  of  time  for  me  to  acquire  a  skill  level  adequate  to  participate  in  the  simulation  was: 
(Circle  one  number  on  the  scale  from  1  to  5.) 


1  =  not  at  all 

2  =  less  than 

3  =  sufficient 

4  =  more  than 

5  =  Totally 

sufficient 

sufficient 

sufficient 

sufficient 

5.  If  you  did  not  have  enough  time  to  become  familiar  with  the  TDS,  how  much  more  time  do  you  believe  you 
would  need? 


6.  How  satisfied  were  you  with  your  overall  performance  while  using  the  CCM  TDS: 


1  =  not  at  all 

2  =  less  than 

3  =  satisfied 

4  =  somewhat 

5  =  Extremely 
satisfied 

satisfied 

satisfied 

satisfied 

7.  When  you  think  about  your  own  knowledge  of  tactical  procedures  and  your  current  skill  level,  how  would 
you  rank  them  now  after  the  TDS  scenario? 


# 

Knowledge  and  skills:  .  Your  current  level 

worst  best 

Familiarity  with  a  five  paragraph  order 

1 

2 

3 

4 

5 

Employ  Machine  guns  in  support  of  Offensive 
Operations 

1 

2 

3 

4 

5 

Employ  Machine  guns  in  support  of  Defensive 
Operations 

1 

2 

3 

4 

5 

Conduct  a  Infantry  Squad/Platoon  Attack 

1 

2 

3 

4 

5 

Control  tactical  movement  of  a  Squad/Platoon  in 
the  Offense 

1 

2 

3 

4 

5 

Conduct  a  Squad/Platoon  Attack  on  Urban  Terrain 

1 

2 

3 

4 

5 

Conduct  a  Squad/Platoon  Defense  on  Urban  Terrain 

1 

2 

3 

4 

5 

Thanks  very  much  for  participating! 
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APPENDIX  D  -  CCM  MEMORY  SHEET  (EXPERIMENT  1) 


Close  Combat  Marines  (CCM) 

Desired  Action 


Selection/Deselect  Units 

Select  Troops  Left  Click  on  Unit 

Unselect  Troops  Left  Click  on  3ncther  part  of  the  screen 


Issue  Orders  Right  Click  on  Unit  to  display  the  orders  menu  >>  Left  Click  on  desired  order 

(Only  orders  that  are  available  will  be  displayed.)  Adot  is  placed  on  the  battlefield  to  indicate  the 
location/destination  of  your  order. 


*  MOVE  FAST 

*  Ml! VI  COVTflT 
FTlR£ 

r  SMOKE 
DEFEND 
F  AMBUSH 

k  DIG  IN 


Move  -Team  members  will  walk 
Move  Fast  -  Team  members  will  run 

Move  Covert  -Team  member  will  crawl  and  then  defend  when  they  arrive  at  destination 
Fire  -  Commands  yourteam  to  commence  firing.  Left  click  ontarget. 

Bright  Green  line:  Yourteam  can  see  fcstarget. 

Dark  Green  line:  Sight  blocked,  but  can  still  shoot. 

Red  line:  Shoot  is  blocked 

Smoke  -  Delivers  smoke  to  a  location  to  screen  movements 
Defend  -Troops  set  into  a  defensive  posture 
Ambush  -  Troops  find  cover  and  fire  only  if  enemy  is  very  close 
Mounl/Dismount  -either  get  into  or  out  of  vehicles 
Dig  -  Troops  dig  and  get  into  a  fighting  hole 


Screen  Navigation 

Use  Map  Monitor  (Thumbnail  Display)  Left  Click  on  the  small  map  to  jump  to  any  location  on  the  battlefield. 
Use  Mouse  and  Keyboard 

Move  Viewpoint  Up  Up  An ow  or  move  cursortotop  edge  of  screen 

Move  Viewpoint  Down  Down  Arow  or  move  cursor  to  bottom  edge  of  screen 
Move  Viewpoint  Right  Right  Arow  or  move  cursorto  right  edge  of  screen 
Move  Viewpoint  Left  Left  Anow  or  move  cursorto  left  edge  of  screen 


Display  Detailed  Terrain  Information  -  Right  click  onterrain  and  HOLD.  Message  will  appear  in  lower 
left  comer. 

Toggle  Trees  on/off  Ctri-rt 


Monitors  and  Toolbars 

(Al  game  information  is  located  at  bottom  of  screen  in  toolbars. 

Turn  them  on  a  off  with  buttons  at  bottom  left  portion  of  the  toolbar) 

You  can: 

Monitor  Morale 

Conduct  Ar  Strikes  with  Fixed  Wing/Rotary  Wing 

(Left  click  on  the  Ar  Icon.then  left  click  on  the  location  where  it  is  to  be  delivered.) 
Conduct  Atillery  Barrage 

(Left  Click  Atillery  Icon  andthen  left  click  where  you  want  ordinance  delivered  on  screen 
Conduct  Mortar  Barrage 

Monitor  Team  Status  (Left  Click  on  teamto  get  individual  team  member's: 

-names 

-health  and  morale 
-billet. 
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APPENDIX  E  -  STATISTICAL  ANALYSIS  (EXPERIMENT  1) 


Experiment  Hypothesis  Test 

H0=  There  is  NO  difference  between  the  methods. 

Ha  =  There  is  a  difference  between  the  methods. 

We  used  F  (ANOVA)  test  for  our  first  experiment.  This  is  because  we  had  3 
groups  of  people.  The  researchers  chose  an  a  =  0.05  as  the  significance  criteria  whether 
to  reject  the  null  hypothesis. 

Since  p  =  0.2241,  which  is  greater  than  a  =  0.05,  we  are  not  able  to  reject  Ho.  We 
conclude  that  there  is  no  significant  difference  between  the  3  methods. 


Methods  vs.  Total  Errors 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

2 

136.50000 

68.2500 

1.1601 

Error 

9 

529.50000 

58.8333 

Prob  >  F 

C.  Total 

11 

666.00000 

0.3563 
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Method  vs.  Total  Error  Factor 


Whole  Model 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

2 

105.42002 

52.7100 

1.7741 

Error 

9 

267.39828 

29.7109 

Prob  >  F 

C.  Total 

11 

372.81829 

0.2241 

102 


Other  Findings 


For  the  following  plots,  we  found  that  there  was  a  significant  correlation  between  Gender 
and  Total  Error  Factor,  as  well  as,  Gaming  Experience  and  Total  Error  Factor. 

Gender  vs.  Total  Error  Factor 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

119.06447 

119.064 

4.6921 

Error 

10 

253.75382 

25.375 

Prob  >  F 

C.  Total 

11 

372.81829 

0.0555 

Gaming  Experience  vs.  Total  Error  Factor 
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Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

177.42417 

177.424 

9.0803 

Error 

10 

195.39412 

19.539 

Prob  >  F 

C.  Total 

11 

372.81829 

0.0130 

Simulation  Experience 


Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

41.15848 

41.1585 

1.2410 

Error 

10 

331.65981 

33.1660 

Prob  >  F 

C.  Total 

11 

372.81829 

0.2913 

Time  In  Service  (TIS) 
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Analysis  of  Variance 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

1 

5.22407 

5.2241 

0.1421 

Error 

10 

367.59423 

36.7594 

Prob  >  F 

C.  Total 

11 

372.81829 

0.7141 
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APPENDIX  F  -  CONSENT  FORM  (EXPERIMENT  2) 


Naval  Postgraduate  School 

Participant  Consent  Form  & 

Minimal  Risk  Statement 

Introduction.  You  are  invited  to  participate  in  a  study  that  will  compare  the  training  effectiveness 
of  a  Tactical  Decision-Making  Simulation  (TDS)  and  a  Tactical  Decision-Making  Game  (TDG). 
This  will  be  conducted  by  the  Naval  Postgraduate  School  Modeling  of  Virtual  Environments  and 
Simulation  (MOVES)  Institute. 

Procedures.  We  will  use  a  lab  in  Watkins  Hall  (212a)  to  conduct  this  experiment  examining 
these  2  different  training  approaches.  The  TDS  that  will  be  used  is  called  Close  Combat  Marine 
(CCM)  version  4.0  and  is  a  virtual  environment  (a  simulation)  used  by  the  USMC  to  train 
Marines  in  tactical  decision-making.  The  researchers  will  provide  the  participants  with  a  briefing 
during  Session  I.  This  will  describe  the  number  of  sessions  and  the  basic  skills  that  will  be 
necessary  to  complete  both  the  TDS  and  the  TDG.  We  will  collect  user  computer  simulation 
experience  and  demographics  in  the  form  of  a  questionnaire  prior  to  Session  I.  Participants  will 
be  divided  into  a  TDG  group  (8  participants)  and  a  TDS  group  (8  participants)  and  remain  in 
those  groups  for  the  duration  of  the  experiment.  Each  of  these  will  be  further  subdivided  into  2 
groups  of  4  each. 

Session  I  TDS  and  TDG  will  be  centered  on  a  tactical  situation  within  a  small  town.  Participants 
will  be  taught  the  necessary  tactics  that  will  be  used  in  the  sessions.  In  the  TDS  session, 
participants  will  be  informed  about  what  the  TDS  is  and  how  to  operate  it.  The  TDG  group  will  be 
introduced  to  the  nature  of  a  TDG  by  participating  in  an  abbreviated  TDG  session.  Both  TDS  and 
TDG  sessions  will  last  approximately  60  minutes  for  both  groups. 

In  Sessions  2,  3,  4,  and  5,  the  TDS  participants  will  go  through  a  different  scenario  each  time.  TDS 
users  will  alternate  between  playing  the  offense  and  the  defense,  changing  roles  after  each  session. 
TDG  users  will  all  be  offense  or  defense  during  a  single  session  and  switch  roles  for  the  next 
session.  The  researchers  will  administer  questionnaires  after  each  session. 

The  researchers  will  record  some  of  the  participants’  actions  using  a  video  camera,  but  will  not  tie 
user  demographic  information  to  the  video.  Participants  will  not  be  identifiable  by  name,  country, 
etc.  The  video  tape  will  be  destroyed  upon  completion  of  the  experiment. 

Risks  and  Benefits.  I  understand  that  this  project  does  not  involve  greater  than  minimal  risk  and 
involves  no  known  reasonably  foreseeable  risks  or  hazards  greater  than  those  encountered  in 
everyday  life.  I  have,  also,  been  informed  of  any  benefits  to  myself  or  to  others  that  may 
reasonably  be  expected  as  a  result  of  this  research. 

Compensation.  I  understand  that  participation  in  this  study  will  earn  me  some  class  credit  in  my 
O A3 101  course  and  that  the  amount  of  my  participation  will  be  provided  to  my  instructor  (Col  L. 
Shattuck).  A  copy  of  the  research  results  will  be  available  at  the  conclusion  of  the  experiment  if  I 
would  like  to  have  a  copy  of  them. 
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Confidentiality  &  Privacy  Act.  I  understand  that  all  records  of  this  study  will  be  kept  confidential 
and  that  my  privacy  will  be  safeguarded.  No  information  will  be  publicly  accessible  which  could 
identify  me  as  a  participant,  and  I  will  be  identified  only  as  a  code  number  on  all  research  forms.  I 
understand  that  records  of  my  participation  will  be  maintained  by  NPS  for  5  years,  after  which  they 
will  be  destroyed. 

Voluntary  Nature  of  the  Study.  I  understand  that  my  participation  is  strictly  voluntary,  and  if  I 
agree  to  participate,  I  am  free  to  withdraw  at  any  time  without  prejudice. 

Points  of  Contact.  I  understand  that  if  I  have  any  questions  or  comments  regarding  this  project 
upon  the  completion  of  my  participation,  I  should  contact  the  Principal  Investigator,  C.  Neil 
Fitzpatrick,  Capt/USMC,  (831)  917-6663,  cnfitzpa@nps.edu,  or  Umit  Ayvaz,  Capt/Turkish  Army, 
uayvaz@nps.edu.  Any  medical  questions  should  be  addressed  to  LTC  Eric  Morgan,  MC,  USA, 
(CO,  POM  Medical  Clinic),  (831)  242-7550,  eric.morgan@nw.amedd.army.mil. 

Statement  of  Consent.  I  have  read  and  understand  the  above  information.  I  have  asked  all 
questions  and  have  had  my  questions  answered.  I  agree  to  participate  in  this  study.  I  will  be 
provided  with  a  copy  of  this  form  for  my  records. 


Participant’s  Name  (please  print) 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 
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APPENDIX  G  -  IRB  REQUEST  (EXPERIMENT  2) 


Amela  Sadagic,  PhD 
C.  Neil  Fitzpatrick,  Capt/USMC 
Umit  Ayvaz,  Capt/Turkish  Army 
MOVES 
Watkins  Hall 

Naval  Postgraduate  School 
Monterey,  California  93943 


asadagic@nps,edu 
(831)  656-3819 

cnfitzpa@nps.edu 
(831)  917-6663 

uayvaz@nps .  edu 
(831)-393-1272 


T o :  Protection  of  Human  Subj  ects  Committee 

Subject:  Comparison  of  the  Training  Effectiveness  of  a  Tactical  Decision-Making  Simulation  (TDS)  and  a 
Tactical  Decision-Making  Game  (TDG). 

Attached  is  a  set  of  documents  outlining  a  proposed  experiment  to  be  conducted  by  C.  Neil  Fitzpatrick 
(Captain  USMC  )  and  Umit  AYVAZ  (Captain  Turkish  Army)  for  their  pilot  study  for  their  thesis  in  a  Lab 
in  Watkins  Hall  (212a). 

1.  We  are  requesting  approval  of  the  described  experimental  protocol.  An  experimental  outline  is 
included  for  your  reference  that  describes  the  methods  and  measures  we  plan  to  use. 

2.  We  include  the  consent  forms,  privacy  act  statements,  all  materials  and  forms  that  a  subject  will  read 
or  fill-out,  and  the  debriefing  forms  (if  applicable)  we  will  be  using  in  the  experiment. 

3.  We  understand  that  any  modifications  to  the  protocol  or  instruments/measures  will  require  submission 
of  updated  IRB  paperwork  and  possible  re-review.  Similarly,  we  understand  that  any  untoward  event 
or  injury  that  involves  a  research  participant  will  be  reported  immediately  to  the  IRB  Chair  and  NPS 
Dean  of  Research. 

The  work  on  this  study  is  a  part  of  NMSO  sponsored  project  (R9D8S)  with  title:  “Studying  the  Synergy  of 
Virtual  Training  Simulations  and  Novel  Training  Methodologies  for  Training  of  the  Ground  Officers,  and 
Evaluation  of  Training  Effectiveness.” 


Dr.  Amela  Sadagic,  Principal  Investigator  (PI) 


C.  Neil  Fitzpatrick,  Student  Researcher 


Umit  AYVAZ,  Student  Researcher 
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APPLICATION  FOR 

HUMAN  SUBJECTS  REVIEW  (HSR) 


HSR  NUMBER  (to  be  assigned) 


PRINCIPAL  INVESTIGATOR(S)  (Full  Name,  Code,  Telephone) 


Dr.  Amela  Sadagic,  MOVES  Research  Associate  Professor,  831.656.3819 


APPROVAL  REQUESTED  [  X  ]  New 

[  ]  Renewal 

LEVEL  OF  RISK  [  ]  Exempt  [  X  ]  Minimal  [  ]  More  than  Minimal 

Justification:  The  subjects  in  the  study  will  be  asked  to  use  virtual  training  simulation  on 
desktop  computer  (they  will  also  use  computer  mouse  and  keyboard)  and  fill  in  several 
questionnaire  forms. 

WORK  WILL  BE  DONE  IN  Watkins  Hall 
Lab  in  Watkins  Hall  (212a) 

ESTIMATED  NUMBER  OF  DAYS  TO 
COMPLETE 

30  days 

MAXIMUM  NUMBER  OF  SUBJECTS 

16 

ESTIMATED  LENGTH  OF  EACH 
SUBJECT’S  PARTICIPATION 

300  minutes  total  (5  sessions  where  1 
session  is  held  on  one  day,  60  min  each 
session) 

SPECIAL  POPULATIONS  THAT  WILL  BE  USED  AS  SUBJECTS 

[  ]  Subordinates  [  ]  Minors  [  X  ]  NPS  Students  [  ]  Special  Needs  (e.g.  Pregnant 
women) 


Specify  safeguards  to  avoid  undue  influence  and  protect  subject’s  rights: 

The  participation  in  the  study  is  voluntary  and  will  not  be  imposed  on  any  NPS  student:  each 
person  will  have  an  option  of  withdrawing  from  the  study  at  any  point  in  time.  The  data 
collected  in  the  study  will  be  safeguarded  as  requested  by  generally  accepted  IRB  standards: 
each  person  will  be  identified  only  as  a  code  number  on  all  research  forms/data  bases;  name 
of  any  person  on  any  signed  document  will  not  be  paired  with  their  code  number  in  order  to 
protect  their  identity;  and  records  of  subject’s  participation  will  be  maintained  by  NPS  for  3 
years,  after  which  they  will  be  destroyed. 


OUTSIDE  COOPERATING  INVESTIGATORS  AND  AGENCIES 
[  1  A  copy  of  the  cooperating  institution’s  HSR  decision  is  attached. 
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TITLE  OF  EXPERIMENT  AND  DESCRIPTION  OF  RESEARCH  (attach  additional  sheet 
if  needed). 

COMPARISON  OF  THE  TRAINING  EFFECTIVENESS  OF  A  TACTICAL  DECISION¬ 
MAKING  SIMULATION  (TPS!  AND  A  TACTICAL  DECISION-MAKING  GAME 
fTDG) 

INTRODUCTION 

We  will  use  the  Lab  in  Watkins  Hall  (212a)  to  conduct  a  study  that  will  compare  the  training 
effectiveness  of  a  Tactical  Decision-Making  Simulation  (TDS)  and  a  Tactical  Decision- 
Making  Game  (TDG).  The  TDS  is  called  Close  Combat  Marine  (CCM)  version  4.0  and  is  a 
virtual  environment  (a  simulation)  used  by  the  USMC  to  train  Marines  in  tactical  decision 
making. 

Participants  will  be  divided  into  a  TDG  group  (8  participants)  and  a  TDS  group  (8 
participants)  and  remain  in  those  groups  for  the  duration  of  the  experiment.  Each  of  these 
will  be  further  subdivided  into  2  groups  of  4  each. 

The  overall  study  will  be  divided  up  into  5  sessions  both  for  the  TDG  and  the  TDS  group: 
SESSION  1 

The  researchers  will  collect  user  computer  simulation  experience  and  demographics  in  the 
form  of  a  questionnaire  prior  to  Session  I.  The  researchers  will  provide  both  TDG  and  TDS 
participants  with  a  briefing  during  Session  I.  This  will  describe  the  number  of  sessions  and 
the  basic  skills  that  will  be  necessary  to  complete  both  the  TDS  and  the  TDG.  In  this 
session,  TDS  and  TDG  will  be  centered  on  a  tactical  situation  within  a  small  town.  In  the 
TDS  session,  participants  will  be  informed  about  what  the  TDS  is  and  how  to  operate  it.  The 
TDG  group  will  be  introduced  to  the  nature  of  a  TDG  by  participating  in  an  abbreviated 
TDG  session. 

Both  TDS  and  TDG  sessions  will  last  approximately  60  minutes. 

SESSION  2.  3.  4.  AND  5: 

The  TDS  participants  will  go  through  a  different  scenario  each  time.  TDS  users  will 
alternate  between  playing  the  offense  and  the  defense,  changing  roles  after  each  session. 
TDG  users  will  all  be  offense  or  defense  during  a  single  session  and  switch  roles  for  the  next 
session.  The  researchers  will  administer  questionnaires  after  each  session,  as  well  as  a  final 
questionnaire  after  Session  5  in  both  TDS  and  TDG.  Both  TDS  and  TDG  sessions  will  last 
approximately  60  minutes. 

The  researchers  will  record  some  of  the  participants’  actions  using  a  video  camera,  but  will 
not  tie  user  demographic  information  to  the  video.  Participants  will  not  be  identifiable  by 
name,  country,  etc.  The  video  tape  will  be  destroyed  upon  completion  of  the  experiment,  but 
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digital  recordings  (values  recorded  by  the  CCM  training  simulation)  will  be  retained  for  the 
purposed  of  training  and  follow-on  work. 

The  researchers  will,  also,  evaluate  each  participant  on  their  ability  to  lead  a  group,  to  make 
tactically  sound  and  timely  decisions,  and  their  situational  awareness  in  the  TDS  and  TDG 
scenarios. 


I  have  read  and  understand  NPS  Notice  on  the  Protection  of  Human  Subjects.  If  there  are  any 
changes  in  any  of  the  above  information  or  any  changes  to  the  attached  Protocol,  Consent 
Form,  or  Debriefing  Statement,  I  will  suspend  the  experiment  until  I  obtain  new  Committee 
approval. 

SIGNATURE _ ,  PI 

SIGNATURE _ ,  Student  Researcher 


SIGNATURE _ ,  Student  Researcher 

DATE 
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APPENDIX  H  -  TDG  SURVEY  (EXPERIMENT  2) 


Pilot  Study  Survey 
-  TDG  Session  1  - 
Pre-questionnaire 

Please  fill  in  the  following  questionnaire.  Answer  all  questions  as  objectively  as  you  can.  All 

information  will  be  held  confidential. 

If  you  need  to  expand  any  answer  please  use  the  reverse  side  of  your  paper  sheet. 

1)  Were  you  required  to  use  computer-based  simulations  as  part  of  your  training  in  the  past?  NO  /  YES 
(circle  one) 

a)  NO. 

b)  If  YES:  What  type  simulations  have  you  used  and  what  skills  were  they  used  to  train? 
i)  Tactical  Level  (Marksmanship  trainer,  ship  driving  simulator,  Language  Trainer,  etc.) 

(1)  Description: 


(2)  Estimate  total  number  of  hours  you  used  it: _ 

(3)  To  gain  competency,  this  amount  of  time  spent  was:  (Circle  one) 


i)  Operational  Level  (OneSAF,  JSAF,  JCATS,  Common  Operational  Picture  of  the  Battlespace,  etc.) 

(1)  Description: 


(2)  Estimate  total  number  of  hours  you  used  it: _ 

(3)  To  gain  competency,  this  amount  of  time  spent  was:  (Circle  one) 
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2)  Do  you  have  experience  playing  commercial  video  games?  NO  /  YES  (circle  one) 

a)  NO. 

b)  If  YES,  What  kind  of  video  games  do  you  play?  (answer  all  that  apply  to  you;  circle  day,  week, 


month  or  year  in  each  line) 

i)  First  person  shooter. 

(Stand  Alone  or  Online) 

year 

ii)  Flight  Simulation. 

(Stand  Alone  or  Online) 

year 

iii)  Racing. 

(Stand  Alone  or  Online) 

year 

iv)  Sports  (Football,  etc.)  (Stand  Alone  or  Online) 

year 

v)  Puzzle,  Card,  Board 

(Stand  Alone  or  Online) 

year 

vi)  Strategy 

(Stand  Alone  or  Online) 

year 

vii)  Adventure,  Fantasy 

(Stand  Alone  or  Online) 

year 

viii)  Arcade 

(Stand  Alone  or  Online) 

year 

ix)  Other 

(Stand  Alone  or  Online) 

year 


Approximately 

hours  per  day  /  week  /  month  / 

Approximately 

hours  per  day  /  week  /  month  / 

Approximately 

hours  per  day  /  week  /  month  / 

Approximately 

hours  per  day  /  week  /  month  / 

Approximately  _ _ 

hours  per  day  /  week  /  month  / 

Approximately _ 

hours  per  day  /  week  /  month  / 

Approximately 

hours  per  day  /  week  /  month  / 

Approximately _ 

hours  per  day  /  week  /  month  / 

Approximately  __ 

hours  per  day  /  week  /  month  / 

3)  Do  you  own  a  personal  computer?  NO  /  YES  (circle  one) 

a)  NO. 

b)  If  YES,  how  often  do  you  use  it?  Approximately _ hours  per  day  /  week  /  month  /  year 

4)  What  hand  do  you  use  to  operate  a  computer  mouse?  LEFT  /  RIGHT  /  I  am  good  with  either  (circle 


5)  How  often  do  you  use  the  following  other  computer  applications?  (please  answer  all  that  apply  to  you;  circle 
day,  week,  month  or  year  in  each  line) 


i)  E-mail: 

ii)  Browse  Web: 

iii)  IMovie: 

iv)  ITunes: 

v)  Excel: 

vi)  Word: 

vii)  Powerpoint: 


Approximately 

Approximately 

Approximately 

Approximately 

Approximately 

Approximately 

Approximately 


hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 


6)  When  you  think  about  your  own  knowledge  of  infantry  tactics,  techniques,  and  procedures,  how  would  you 
rank  them?  (please  circle  NA  or  one  number  between  1  and  7  in  each  line) 
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# 

Knowledge  and  skills:  Your  current  level 

Poor 

Excellent 

1. 

Familiarity  with  a  five  paragraph  order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ  Machine  guns  in  support  of  Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Employ  Machine  guns  in  support  of  Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Infantry  Offensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Infantry  Defensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical  Movement  On  Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire  and  Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Infantry  Attack  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Infantry  Defense  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue  an  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct  Tactical  Movement  in  the  Offense 

NA 

1 

2 

3 

4 

5 

6 

7 

7)  Subject#  (your  code  name): _ 

8)  Date: _  Time: 

9)  Year  of  birth:  _ 

10)  Country  of  Citizenship: _ 

11)  Service  component:  (Circle  one) 


Army 

Navy 

Air  Force 

Marine  Corps 

Coast  Guard 

Civilian 

12)  Briefly  describe  your  Primary  MOS,  (Example:  Infantry,  Surface  Warfare,  communications,  etc.): 


13)  NPS  Curriculum: _ 

14)  Your  Rank:  _ 

15)  Time  in  Service: _ years _ months 

Thank  you  very  much  for  participating  in  the  study. 


115 


Pilot  Study  Survey 
-  TDG  Session  2  - 
Questionnaire 


Please  fill  in  the  following  questionnaire.  Answer  all  questions  as  objectively  as  you  can. 
All  information  will  be  held  in  confidence. 

If  you  need  to  expand  any  answer  please  use  the  reverse  side  of  your  paper. 


Participant  #  =  (Your  code  name) : _ 

Date: _  Time: _ 

1)  What  training  group  were  you  in?  (Circle  One) 

a.  Offense 

b.  Defense 

2)  For  each  person  (including  yourself)  give  a  score  between  0  and  100  which  estimates  the  degree  to  which  that 
person  was  the  ‘leader’  in  the  last  training  session.  (The  four  scores  must  add  to  100). 

(For  example,  if  no  clear  leader  emerged,  then  each  should  have  a  score  of  25.  If  Person  X  played  the  major  role 
in  this  regard,  then  Person  X  would  have  a  score  of  say  80,  and  the  other  2  people,  a  score  of  10  each.  A  score 
of  0  means  that  the  corresponding  person  was  in  no  way  a  leader;  whereas  a  score  of  100  means  that  the 
corresponding  person  was  the  clear  and  only  leader.) 


Person  (code  name): 

Score/100: 

Total  Score: 

100 

3)  Think  about  a  previous  time  when  you  enjoyed  working  together  in  a  group.  To  what  extent  have  you  enjoyed 
the  group  experience  just  now? 

I  enjoyed  the  group  experience  just  now  (please  check  one  number  between  1  and  7): 


1.  Not  at  all 

It  was  nothing  like  the  previous  experience  of  working  in  a  group. 

2. 

3. 

4. 

5. 

6. 

7.  Very  much  so. 

I  enjoyed  it  just  as  much  as  the  previous  experience  of  working  in  a  group. 

4)  Think  back  now  about  the  session  that  you  have  just  completed.  For  example,  to  what  extent  in  your 
imagination  can  you  move  around  that  terrain  (battlefield)  now? 


I  can  think  myself  back  on  that  terrain/battlefield  (please  check  one  number  between  1  and  7): 
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1.  Not  at  all 

I  cannot  imagine  now  at  all  being  on  that  terrain/battlefield  now. 

2. 

3. 

5. 

6. 

7.  Very  much  so. 

I  can  strongly  imagine  being  back  on  that  terrain/battlefield  now. 

5)  In  the  last  session,  to  what  extent  did  you  have  the  sense  of  the  other  three  people  being  together  with  you? 


In  the  last  session,  I  had  a  sense  that  the  other  three  people  were  with  me  (please  check  one  number  between  1 
and  7): 


1.  Not  at  all 

I  did  not  have  a  sense  that  the  other  three  people  were  with  me. 

2. 

3. 

4. 

5. 

6. 

7.  Very  much  so. 

The  other  three  people  were  very  much  with  me. 

6)  Who  did  most  of  the  talking?  Give  a  score  to  each  person  between  0  and  1 00,  where  the  four  scores  add  to 
100.  (A  person  would  be  given  a  score  of  near  100  only  if  they  did  almost  all  the  talking.  They  would  be  given  a 
score  of  near  0  if  they  did  almost  no  talking). 

The  extent  to  which  each  person  did  most  of  the  talking  was: 


Person  (code  name): 

Score/100: 

Total  Score: 

100 

7)  Overall,  how  cooperative  was  each  of  the  other  three  people  (please  check  one  value  between  1  and  7  for 
each  person). 


Person  (code  name) 

■) 

1.  Not  at  all 

S/he  was  not  cooperative  at  all 

2. 

3. 

4. 

5. 

6. 

7.  Very  much  so. 

S/he  was  very  cooperative 

8)  Think  back  now  about  past  session,  what  were  good  tactical  decision  and  procedures  done  by  your  team? 
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9)  Think  back  now  about  past  session,  what  were  bad  tactical  decision  and  procedures  done  by  your  team? 


10)  Think  back  now  about  past  session,  what  were  good  tactical  decision  and  procedures  done  by  other  team? 


11)  Think  back  now  about  past  session,  what  were  bad  tactical  decision  and  procedures  done  by  other  team? 
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12)  What  can  you  say  about: 


a)  Team  Member  #1,  (Participant’s  code  name: _ )  where  Poor  (1)  -  Excellent  (7). 

(please  circle  NA  or  one  number  between  1  and  7  in  each  line) 


# 

Knowledge  and  skills: 

.  Your  current  level 

poor 

Excellent 

1. 

Understood  Operations  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified  Enemy  Most  Probably  Course  of  Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created  a  Scheme  of  Maneuver  (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers  Adjacent  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Created  a  Fire  Support  Plan  (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified  Secondary  MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered  the  following  in  Mission  Planning 
Process: 

7. a 

-  Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

-  Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.c 

-  Troops  Available 

NA 

1 

2 

3 

4 

5 

6 

7 

7.d 

-  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

7.e 

-  Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Identified  Enemy  of  Center  of  Gravity  and 

Attacks  It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted  to  Enemy  Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took  Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Maintained  Oversight  of  All  Subordinate  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

12. 

Communicated  His  Plan/Intent  to  Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

13. 

Ensured  that  Subordinates  what  they  had  been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 
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# 

Knowledge  and  skills: 

.  Your  current  level 

poor 

Excellent 

14. 

Used  deception  and  creativity  to  outwit  the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated  from  doctrine  when  necessary 

NA 

i 

2 

3 

4 

5 

6 

7 

16. 

Enemy  Location 

NA 

i 

2 

3 

4 

5 

6 

7 

17. 

Size  of  Enemy  Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Direction  of  Enemy  Movement 

NA 

1 

2 

3 

4 

5 

6 

7 

19. 

Understanding  of  Battle  Space  Geometry 

NA 

1 

2 

3 

4 

5 

6 

7 

b)  Team  Member  #2,  (Participant’s  code  name: _ )  where  Poor  (1)  -  Excellent  (7). 

(please  circle  NA  or  one  number  between  1  and  7  in  each  line) 


# 

Knowledge  and  skills: 

.  Your  current  level 

poor 

Excellent 

1. 

Understood  Operations  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified  Enemy  Most  Probably  Course  of  Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created  a  Scheme  of  Maneuver  (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers  Adjacent  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Created  a  Fire  Support  Plan  (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified  Secondary  MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered  the  following  in  Mission  Planning 
Process: 

7. a 

-  Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

-  Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.c 

-  Troops  Available 

NA 

1 

2 

3 

4 

5 

6 

7 

7.d 

-  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

7.e 

-  Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Identified  Enemy  of  Center  of  Gravity  and 

Attacks  It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted  to  Enemy  Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took  Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Maintained  Oversight  of  All  Subordinate  Units 

NA 

i 

2 

3 

4 

5 

6 

7 

12. 

Communicated  His  Plan/Intent  to  Subordinates 

NA 

i 

2 

3 

4 

5 

6 

7 

13. 

Ensured  that  Subordinates  what  they  had  been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used  deception  and  creativity  to  outwit  the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 
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Knowledge  and  skills: 

.  Your  current  level 

poor 

Excellent 

15. 

Deviated  from  doctrine  when  necessary 

NA 

1 

2 

3 

4 

5 

6 

7 

16. 

Enemy  Location 

NA 

1 

2 

3 

4 

5 

6 

7 

17. 

Size  of  Enemy  Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Direction  of  Enemy  Movement 

NA 

1 

2 

3 

4 

5 

6 

7 

19. 

Understanding  of  Battle  Space  Geometry 

NA 

1 

2 

3 

4 

5 

6 

7 

c)  Team  Member  #3,  (Participant’s  code  name: _ ),  where  Poor  (1)  -  Excellent  (7). 

(please  circle  NA  or  one  number  between  1  and  7  in  each  line) 


# 

Knowledge  and  skills: 

.  Your  current  level 

poor 

Excellent 

1. 

Understood  Operations  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified  Enemy  Most  Probably  Course  of  Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created  a  Scheme  of  Maneuver  (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers  Adjacent  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Created  a  Fire  Support  Plan  (FSP) 

NA 

i 

2 

3 

4 

5 

6 

7 

6. 

Identified  Secondary  MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered  the  following  in  Mission  Planning 
Process: 

7. a 

-  Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

-  Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.c 

-  Troops  Available 

NA 

i 

2 

3 

4 

5 

6 

7 

7.d 

-  Terrain 

NA 

i 

2 

3 

4 

5 

6 

7 

7.e 

-  Time 

NA 

1 

2 

3 

_ 

4 

5 

6 

7 

8. 

Identified  Enemy  of  Center  of  Gravity  and 

Attacks  It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted  to  Enemy  Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took  Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Maintained  Oversight  of  All  Subordinate  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

12. 

Communicated  His  Plan/Intent  to  Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

13. 

Ensured  that  Subordinates  what  they  had  been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used  deception  and  creativity  to  outwit  the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated  from  doctrine  when  necessary 

NA 

1 

2 

3 

4 

5 

6 

7 

1 _ 
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# 

Knowledge  and  skills: 

.  Your  current  level 

poor 

Excellent 

16. 

Enemy  Location 

NA 

i 

2 

3 

4 

5 

6 

7 

17. 

Size  of  Enemy  Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Direction  of  Enemy  Movement 

NA 

i 

2 

3 

4 

5 

6 

7 

19. 

Understanding  of  Battle  Space  Geometry 

NA 

i 

2 

3 

4 

5 

6 

7 

13)  YOURSELF  on  the  following  criteria  where  Poor  (1)  -  Excellent  (7).  (please  circle  NA  or  one  number 
between  1  and  7  in  each  line) 


# 

Knowledge  and  skills: 

.  Your  current  level 

poor 

Excellent 

1. 

Understood  Operations  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified  Enemy  Most  Probably  Course  of  Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created  a  Scheme  of  Maneuver  (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers  Adjacent  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Created  a  Fire  Support  Plan  (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified  Secondary  MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered  the  following  in  Mission  Planning 
Process: 

7. a 

-  Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

-  Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.c 

-  Troops  Available 

NA 

1 

2 

3 

4 

5 

6 

7 

7.d 

-  Terrain 

NA 

i 

2 

3 

4 

5 

6 

7 

7.e 

-  Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Identified  Enemy  of  Center  of  Gravity  and 

Attacks  It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted  to  Enemy  Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took  Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Maintained  Oversight  of  All  Subordinate  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

12. 

Communicated  His  Plan/Intent  to  Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

13. 

Ensured  that  Subordinates  what  they  had  been 
told 

NA 

1 

2 

3 

_ 

4 

5 

6 

7 

14. 

Used  deception  and  creativity  to  outwit  the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated  from  doctrine  when  necessary 

NA 

1 

2 

3 

4 

5 

6 

7 

16. 

Knew  Enemy  Location 

NA 

1 

2 

3 

4 

5 

6 

7 

1 _ 
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# 

Knowledge  and  skills: 

.  Your  current  level 

poor 

Excellent 

17. 

Knew  the  Size  of  the  Enemy  Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Knew  the  Direction  of  the  Enemy's  Movement 

NA 

1 

2 

3 

4 

5 

6 

7 

19. 

Understood  the  Battle  Space  Geometry 

NA 

i 

2 

3 

4 

5 

6 

7 

14)  If  you  did  not  have  enough  time  to  become  familiar  with  the  infantry  tactics  of  the  offense  or  defense 
(whichever  you  participated  in),  how  much  more  time  do  you  believe  you  would  need? 


15)  How  satisfied  were  you  with  your  overall  performance  while  participating  in  the  TDG: 


Thanks  very  much  for  participating! 

NOTE:  Questionnaires  for  Sessions  3  -  5  are  identical  to  Session  2  (above). 
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Pilot  Study  Survey 
-  TDG  FINAL  - 


Questionnaire 

Please  fill  in  the  following  questionnaire.  Answer  all  questions  as  objectively  as  you  can. 
All  information  will  be  held  in  confidence. 

If  you  need  to  expand  any  answer  please  use  the  reverse  side  of  your  paper  sheet. 

1)  Which  of  the  following  items/drills  did  you  exercise  at  least  once  during  the  TDG  sessions?  (Circle  all  that 
apply  from  the  list  below.) 


# 

Knowledge  and  skills: 

1. 

Familiarity  with  a  five  paragraph  order 

2. 

Employ  Machine  guns  in  support  of  Offensive  Operations 

3. 

Employ  Machine  guns  in  support  of  Defensive  Operations 

4. 

Infantry  Offensive  Operations  in  Field 

5. 

[Infantry  Defensive  Operations  in  Field 

6. 

Tactical  Movement  On  Objective 

7. 

Fire  and  Maneuver 

8. 

Infantry  Attack  in  Urban  Terrain 

9. 

Infantry  Defense  in  Urban  Terrain 

10. 

Issue  an  Order 

11. 

Conduct  Tactical  Movement  in  the  Offense 

2)  Of  the  actions  that  you  chose  in  the  previous  question  (Question  1),  with  which  did  you  have  the  most 
difficulty? 

(circle  three) 


# 

Knowledge  and  skills: 

1. 

Familiarity  with  a  five  paragraph  order 

2. 

Employ  Machine  guns  in  support  of  Offensive  Operations 

3. 

Employ  Machine  guns  in  support  of  Defensive  Operations 

4. 

Infantry  Offensive  Operations  in  Field 

5. 

Infantry  Defensive  Operations  in  Field 

6. 

Tactical  Movement  On  Objective 
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# 

Knowledge  and  skills: 

7. 

Fire  and  Maneuver 

8. 

Infantry  Attack  in  Urban  Terrain 

9. 

Infantry  Defense  in  Urban  Terrain 

10. 

Issue  an  Order 

11. 

Conduct  Tactical  Movement  in  the  Offense 

3)  Of  the  actions  that  you  chose  in  the  Question  1 ,  which  do  you  feel  you  improved  upon  most  during  the  TDG 

sessions?  (circle  three) 


# 

Knowledge  and  skills: 

1. 

Familiarity  with  a  five  paragraph  order 

2. 

Employ  Machine  guns  in  support  of  Offensive  Operations 

3. 

Employ  Machine  guns  in  support  of  Defensive  Operations 

4. 

Infantry  Offensive  Operations  in  Field 

5. 

Infantry  Defensive  Operations  in  Field 

6. 

Tactical  Movement  On  Objective 

7. 

Fire  and  Maneuver 

8. 

Infantry  Attack  in  Urban  Terrain 

9. 

Infantry  Defense  in  Urban  Terrain 

10. 

Issue  an  Order 

11. 

Conduct  Tactical  Movement  in  the  Offense 

4)  When  you  think  about  your  own  knowledge  of  infantry  tactics,  techniques,  and  procedures,  how  would  you  rank 
them?  (please  circle  NA  or  one  number  between  1  and  7  in  each  line) 


# 

Knowledge  and  skills:  Your  current 

level 

Poor 

Excellent 

1. 

Familiarity  with  a  five  paragraph  order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ  Machine  guns  in  support  of  Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

4. 

5. 

Employ  Machine  guns  in  support  of  Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

Infantry  Offensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

Infantry  Defensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical  Movement  On  Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire  and  Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 
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# 

Knowledge  and  skills: 

Your  current 

level 

Poor 

Excellent 

8. 

Infantry  Attack  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

- 

9. 

Infantry  Defense  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue  an  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct  Tactical  Movement  in  the  Offense 

NA 

1 

2 

3 

4 

5 

_ 

6 

7 

5)  When  you  think  about  your  own  knowledge  of  infantry  tactics,  techniques,  and  procedures,  how  much  did 
you  improve  in  each?  (please  circle  NA  or  one  number  between  1  and  7  in  each  line) 


# 

Knowledge  and  skills: 

They 

Improved 

Improve 

greatly 

Your  improvement 

did 

at 

not 

all 

1. 

Familiarity  with  a  five  paragraph  order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ  Machine  guns  in  support  of  Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Employ  Machine  guns  in  support  of  Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Infantry  Offensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Infantry  Defensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical  Movement  On  Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire  and  Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Infantry  Attack  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Infantry  Defense  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue  an  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct  Tactical  Movement  in  the  Offense 

NA 

1 

2 

3 

4 

5 

6 

7 

4)  Rate  the  TDG  experience  on  its  ability  to  provide  you  with  a  good  mental  picture  of  the  battle  space: 


1  =  poor 

2 

3 

4  =  average 

5 

6 

7  =  excellent 

5)  Rate  the  following  statements  (circle  one  number  for  each  statement): 


i. 

“During  the  TDG  sessions,  I  felt  like  I  was 
taking  part  in  training.” 

1  = 

strongly 

disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 

2. 

“During  the  TDG  sessions,  I  felt  like  my 
actions/my  plan  had  no  consequences  to 
the  plans  of  the  group.” 

1  = 

strongly 

disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 

3. 

“During  the  TDG  sessions,  I  felt  like  I  was 
part  of  the  group  working  together.” 

1  = 

strongly 

disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 
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4. 

“During  the  TDG  sessions,  I  felt  like  I  was 
playing  a  game.” 

1  = 

strongly 

disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 

5. 

“During  the  TDG  sessions,  I  felt  isolated 
from  the  others.” 

1  = 

strongly 

disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 

6)  In  your  opinion  what  was  the  overall  quality  of  training  in  the  TDG?  (circle  one) 


1  =  not  satisfactory 

2 

3 

4  =  average 

5 

6 

7  =  excellent 

7)  Throughout  all  the  TDG  sessions 

,  do  you  believe  that  you  made  good  tactical  decisions?  (circle  one  number) 

1  =  poor 

2 

3 

4  =  satisfactorily 

5 

6 

7  =  excellent 

8)  How  well  did  the  TDG  sessions  meet  your  expectations  regarding  training  value? 


1  =  not  satisfactorily 

2 

3 

4  =  satisfactorily  5 

6 

7  =  exceptionally 

well 

9)  When  you  think  about  your  future  training  needs,  to  what  extent  would  you  like  it  to  be  supplemented  with 
TDGs? 


1  =  not  satisfactorily 

2 

3 

4  =  satisfactorily 

5 

6 

7  =  exceptionally 
well 

10)  When  you  think  about  your  future  training  needs,  to  what  extent  would  you  like  it  to  be  supplemented  with 
computer  based  simulation  of  a  TDG? 


1  =  not  satisfactorily 

2 

3 

4  =  satisfactorily 

5 

6  7  =  exceptionally 

| _ | _ well 

Thank  you  very  much  for  participating  in  the  study. 
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APPENDIX  I  -  TDS  SURVEY  (EXPERIMENT  2) 


Pilot  Survey 
-  TDS  Session  1  - 
Pre-questionnaire 

Please  fill  in  the  following  questionnaire.  Answer  all  questions  as  objectively  as  you  can.  All 

information  will  be  held  confidential. 

If  you  need  to  expand  any  answer  please  use  the  reverse  side  of  your  paper  sheet. 

1)  Were  you  required  to  use  computer-based  simulations  as  part  of  your  training  in  the  past?  NO  /  YES  (circle 
one) 

a)  NO. 

b)  If  YES:  What  type  simulations  have  you  used  and  what  skills  were  they  used  to  train? 
i)  Tactical  Level  (Marksmanship  trainer,  ship  driving  simulator,  Language  Trainer,  etc.) 

(1)  Description: 


(2)  Estimate  total  number  of  hours  you  used  it: _ 

(3)  To  gain  competency,  this  amount  of  time  spent  was:  (Circle  one) 


ii)  Operational  Level  (OneSAF,  JSAF,  JCATS,  Common  Operational  Picture  of  the  Battlespace,  etc.) 
Description: 


Estimate  total  number  of  hours  you  used  it: _ 

(1)  To  gain  competency,  this  amount  of  time  spent  was:  (Circle  one) 


2)  Do  you  have  experience  playing  commercial  video  games?  NO  /  YES  (circle  one) 
a)  NO. 
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b)  If  YES,  What  kind  of  video  games  do  you  play?  (please  answer  all  that  apply  to  you;  circle  day, 


week,  month  or  year  in  each  line) 

i)  First  person  shooter.  (Stand  Alone  or  Online) 
year 

ii)  Flight  Simulation.  (Stand  Alone  or  Online) 
year 

iii)  Racing.  (Stand  Alone  or  Online) 


Approximately  _ _ hours  per  day  /  week  /  month  / 

Approximately  _ _ hours  per  day  /  week  /  month  / 

Approximately _ hours  per  day  /  week  /  month  / 


year 

iv)  Sports  (Football,  etc.)  (Stand  Alone  or  Online) 
year 

v)  Puzzle,  Card,  Board  (Stand  Alone  or  Online) 
year 

vi)  Strategy  (Stand  Alone  or  Online) 

year 

vii)  Adventure,  Fantasy  (Stand  Alone  or  Online) 
year 

viii)  Arcade  (Stand  Alone  or  Online) 

year 

ix)  Other  (Stand  Alone  or  Online) 

year 


Approximately 

hours  per  day  /  week  /  month  / 

Approximately 

hours  per  day  /  week  /  month  / 

Approximately 

hours  per  day  /  week  /  month  / 

Approximately 

hours  per  day  /  week  /  month  / 

Approximately _ _ 

hours  per  day  /  week  /  month  / 

Approximately _ 

hours  per  day  /  week  /  month  / 

3) 


4) 

one) 


Do  you  own  a  personal  computer?  NO  /  YES  (circle  one) 

a)  NO. 

b)  If  YES,  how  often  do  you  use  it?  Approximately _ hours  per  day  /  week  /  month  /  year 

What  hand  do  you  use  to  operate  a  computer  mouse?  LEFT  /  RIGHT  /  I  am  good  with  either  (circle 


5)  How  often  do  you  use  the  following  other  computer  applications?  (please  answer  all  that  apply  to  you;  circle 
day,  week,  month,  or  year  in  each  line) 


i)  E-mail: 

ii)  Browse  Web: 

iii)  IMovie: 

iv)  ITunes: 

v)  Excel: 

vi)  Word: 

vii)  Powerpoint: 


Approximately 

Approximately 

Approximately 

Approximately 

Approximately 

Approximately 

Approximately 


hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 
hours  per  day  /  week  /  month  /  year 


6)  When  you  think  about  your  own  knowledge  of  infantry  tactics,  techniques,  and  procedures,  how  would  you 
rank  them?  (please  circle  NA  or  one  number  between  1  and  7  in  each  line) 


# 

Knowledge  and  skills: 

Your  current  level 

poor 

Excellent 

1. 

Familiarity  with  a  five  paragraph  order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ  Machine  guns  in  support  of  Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 
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3. 

Employ  Machine  guns  in  support  of  Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Infantry  Offensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Infantry  Defensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical  Movement  On  Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire  and  Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Infantry  Attack  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Infantry  Defense  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue  an  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct  Tactical  Movement  in  the  Offense 

NA 

1 

2 

3 

4 

5 

6 

7 

1)  Subject#  =  (your  code  name) : _ 

2)  Date: _  Time: 

3)  Year  of  birth:  _ 

4)  Country  of  Citizenship: _ 

5)  Service  component:  (Circle  one) : 


Army 

Navy 

Air  Force 

Marine  Corps 

Coast  Guard 

Civilian 

6)  Briefly  describe  your  Primary  MOS,  (Example:  Infantry,  Surface  Warfare,  communications,  etc.): 


7)  NPS  Curriculum: _ 

8)  Your  Rank:  _ 

9)  Time  in  Service: _ years _ months 

Thank  you  very  much  for  participating  in  the  study. 
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Pilot  Study  Survey 
-  TDS  Session  2  - 


Platoon  Leader  Questionnaire 

Please  fill  in  the  following  questionnaire.  Answer  all  questions  as  objectively  as  you  can. 

All  information  will  be  held  confidential. 

If  you  need  to  expand  any  answer  please  use  the  reverse  side  of  your  paper  sheet. 


Participant  #  =  (Your  code  name)  : _ 

Date: _  Time: _ 

1)  What  training  group  were  you  in:  (Circle  One) 

a.  Offense 

b.  Defense 

2)  For  each  person  (including  yourself)  give  a  score  between  0  and  100  which  estimates  the  degree  to  which  that 
person  was  the  ‘leader’  in  the  last  training  session.  (The  four  scores  must  add  to  100). 

(For  example,  if  no  clear  leader  emerged,  then  each  should  have  a  score  of  25.  If  Person  X  played  the  major  role 
in  this  regard,  then  Person  X  would  have  a  score  of  say  80,  and  the  other  2  people,  a  score  of  10  each.  A  score 
of  0  means  that  the  corresponding  person  was  in  no  way  a  leader,  whereas  a  score  of  100  means  that  the 
corresponding  person  was  the  clear  and  only  leader.) 


Person  (code  name): 

Score/100: 

Total  Score: 

100 

3)  Think  about  a  previous  time  when  you  enjoyed  working  together  in  a  group.  To  what  extent  have  you  enjoyed 
the  group  experience  just  now? 


I  enjoyed  the  group  experience  just  now  (please  check  one  number  between  1  and  7): 


1.  Not  at  all 

It  was  nothing  like  the  previous  experience  of  working  in  a  group. 

2. 

3. 

5. 

6. 

7.  Very  much  so. 

I  enjoyed  it  just  as  much  as  the  previous  experience  of  working  in  a  group. 

4)  Think  back  now  about  the  session  that  you  have  just  completed.  For  example,  to  what  extent  in  your 
imagination  can  you  move  around  that  terrain  (battlefield)  now? 


I  can  think  myself  back  on  that  terrain/battlefield  (please  check  one  number  between  1  and  7): 


1.  Not  at  all 

I  cannot  imagine  now  at  all  being  on  that  terrain/battlefield  now. 

2. 

3. 

4. 
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5. 

6. 

7.  Very  much  so. 

I  can  strongly  imagine  being  back  on  that  terrain/battlefield  now. 

5)  In  the  last  session,  to  what  extent  did  you  have  the  sense  of  the  other  three  people  being  together  with  you? 


In  the  last  session,  I  had  a  sense  that  the  other  three  people  were  with  me  (please  check  one  number  between  1 
and  7): 


1.  Not  at  all 

I  did  not  have  a  sense  that  the  other  three  people  were  with  me. 

2. 

3. 

4. 

5. 

6. 

7.  Very  much  so. 

The  other  three  people  were  very  much  with  me. 

6)  Who  did  most  of  the  talking?  Give  a  score  to  each  person  between  0  and  1 00,  where  the  four  scores  add  to 
100.  (A  person  would  be  given  a  score  of  near  100  only  if  they  did  almost  all  the  talking.  They  would  be  given  a 
score  of  near  0  if  they  did  almost  no  talking). 


The  extent  to  which  each  person  did  most  of  the  talking  was: 


Person  (code  name): 

Score/100: 

Total  Score: 

100 

7)  Overall,  how  cooperative  was  each  of  the  other  three  people  (please  check  one  value  between  1  and  7  for 
each  person). 


Person  (code  name) 

■) 

1.  Not  at  all 

S/he  was  not  cooperative  at  all 

2. 

3. 

4. 

5. 

6. 

7.  Very  much  so. 

S/he  was  very  cooperative 

8)  Think  back  now  about  past  session,  what  were  good  tactical  decision  and  procedures  done  by  your  team? 
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9)  Think  back  now  about  past  session,  what  were  bad  tactical  decision  and  procedures  done  by  your  team? 


10)  Think  back  now  about  past  session,  what  were  good  tactical  decision  and  procedures  done  by  other  team? 


11)  Think  back  now  about  past  session,  what  were  bad  tactical  decision  and  procedures  done  by  other  team? 
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12)  What  can  you  say  about  your  SQUAD  LEADERS  on  the  following  points,  where  Poor  (1)  -  Excellent  (7). 
(please  circle  NA  or  one  number  between  1  and  7  in  each  line) 

a)  Squad  Leader  #  1 ,  Participant’ s  code  name : _ 


# 

Knowledge  and  skills: 

poor 

Excellent 

Your  current  level 

1. 

Understood  Operations  Order 

NA 

i 

2 

3 

4 

5 

6 

7 

2. 

Identified  Enemy  Most  Probably  Course  of  Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created  a  Scheme  of  Maneuver  (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers  Adjacent  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Created  a  Fire  Support  Plan  (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified  Secondary  MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered  the  following  in  Mission  Planning 
Process: 

7. a 

-  Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

-  Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.c 

-  Troops  Available 

NA 

1 

2 

3 

4 

5 

6 

7 

7.d 

-  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

7.e 

-  Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Identified  Enemy  of  Center  of  Gravity  and 

Attacks  It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted  to  Enemy  Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took  Charge 

NA 

1 

_ 

2 

3 

4 

5 

6 

7 

11. 

Maintained  Oversight  of  All  Subordinate  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

12. 

Communicated  His  Plan/Intent  to  Subordinates 

NA 

i 

2 

3 

4 

5 

6 

7 

13. 

Ensured  that  Subordinates  what  they  had  been 
told 

NA 

i 

2 

3 

4 

5 

6 

7 

14. 

Used  deception  and  creativity  to  outwit  the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated  from  doctrine  when  necessary 

NA 

1 

2 

3 

4 

5 

6 

7 

16. 

Enemy  Location 

NA 

1 

2 

3 

4 

5 

6 

7 

1 _ 
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# 

Knowledge  and  skills: 

poor 

Excellent 

Your  current  level 

17. 

Size  of  Enemy  Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Direction  of  Enemy  Movement 

NA 

1 

2 

3 

4 

5 

6 

7 

19. 

Understanding  of  Battle  Space  Geometry 

NA 

1 

2 

3 

4 

5 

6 

7 

b)  Squad  Leader  #2,  Participant’s  code  name: 


# 

Knowledge  and  skills: 

poor 

Excellent 

Your  current  level 

1. 

Understood  Operations  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified  Enemy  Most  Probably  Course  of  Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created  a  Scheme  of  Maneuver  (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers  Adjacent  Units 

NA 

i 

2 

3 

4 

5 

6 

7 

5. 

Created  a  Fire  Support  Plan  (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified  Secondary  MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered  the  following  in  Mission  Planning 
Process: 

7. a 

-  Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

-  Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.c 

-  Troops  Available 

NA 

1 

2 

3 

4 

5 

6 

7 

7.d 

-  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

7.e 

-  Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Identified  Enemy  of  Center  of  Gravity  and 

Attacks  It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted  to  Enemy  Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took  Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Maintained  Oversight  of  All  Subordinate  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

12. 

Communicated  His  Plan/Intent  to  Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

13. 

Ensured  that  Subordinates  what  they  had  been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used  deception  and  creativity  to  outwit  the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated  from  doctrine  when  necessary 

NA 

1 

2 

3 

4 

5 

6 

7 

16. 

Enemy  Location 

NA 

1 

2 

3 

4 

5 

6 

7 

17. 

Size  of  Enemy  Element 

NA 

1 

2 

3 

4 

5 

6 

7 
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Knowledge  and  skills: 

poor 

Excellent 

Your  current  level 

18. 

Direction  of  Enemy  Movement 

NA 

1 

2 

3 

4  5 

6  7 

19. 

Understanding  of  Battle  Space  Geometry 

NA 

1 

2 

3 

4  5 

6  7 

c)  Squad  Leader  #3,  Participant’s  code  name: 


# 

Knowledge  and  skills: 

poor 

Excellent 

Your  current  level 

1. 

Understood  Operations  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified  Enemy  Most  Probably  Course  of  Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created  a  Scheme  of  Maneuver  (SOM) 

NA 

i 

2 

3 

4 

5 

6 

7 

4. 

Considers  Adjacent  Units 

NA 

i 

2 

3 

4 

5 

6 

7 

5. 

Created  a  Fire  Support  Plan  (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified  Secondary  MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered  the  following  in  Mission  Planning 
Process 

7. a 

-  Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

7.b 

-  Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.c 

-  Troops  Available 

NA 

1 

2 

3 

4 

5 

6 

7 

7.d 

-  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

7.e 

-  Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

Identified  Enemy  of  Center  of  Gravity  and 

Attacks  It. 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Adjusted  to  Enemy  Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took  Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Maintained  Oversight  of  All  Subordinate  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

12. 

Communicated  His  Plan/Intent  to  Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

13. 

Ensured  that  Subordinates  what  they  had  been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used  deception  and  creativity  to  outwit  the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated  from  doctrine  when  necessary 

NA 

1 

2 

3 

4 

5 

6 

7 

16. 

Enemy  Location 

NA 

1 

2 

3 

4 

5 

6 

7 

17. 

Size  of  Enemy  Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Direction  of  Enemy  Movement 

NA 

i 

2 

3 

4 

5 

6 

7 
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#  Knowledge  and  skills:  Your  current  level 


poor 

Excellent 


19. 

Understanding  of  Battle  Space  Geometry 

NA 

1 

2 

3 

4 

5 

6 

7 

d)  YOURSELF  on  the  following  criteria  where  Poor  (1)  -  Excellent  (7).  (please  circle  NA  or  one 

number  between  1  and  7  in  each  line) 

# 

Knowledge  and  skills: 

Your  current  level 

poor 

Excellent 

1. 

Understood  Operations  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Identified  Enemy  Most  Probably  Course  of  Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Created  a  Scheme  of  Maneuver  (SOM) 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Considers  Adjacent  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Created  a  Fire  Support  Plan  (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Identified  Secondary  MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Considered  the  following  in  Mission  Planning 
Process: 

7. a 

-  Mission 

NA 

i 

2 

3 

4 

5 

6 

7 

7.b 

-  Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

7.c 

-  Troops  Available 

NA 

1 

2 

3 

4 

5 

6 

7 

7.d 

-  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

7.e 

-  Time 

NA 

1 

2 

3 

4 

5 

6 

7 

8. 

9. 

Identified  Enemy  of  Center  of  Gravity  and 

Attacks  It. 

NA 

1 

2 

3 

4 

5 

6 

7 

Adjusted  to  Enemy  Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Took  Charge 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

12. 

Maintained  Oversight  of  All  Subordinate  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

Communicated  His  Plan/Intent  to  Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

13. 

Ensured  that  Subordinates  what  they  had  been 
told 

NA 

1 

2 

3 

4 

5 

6 

7 

14. 

Used  deception  and  creativity  to  outwit  the 
enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

15. 

Deviated  from  doctrine  when  necessary 

NA 

1 

2 

3 

4 

5 

6 

7 

16. 

Enemy  Location 

NA 

1 

2 

3 

4 

5 

6 

7 

17. 

Size  of  Enemy  Element 

NA 

1 

2 

3 

4 

5 

6 

7 

18. 

Direction  of  Enemy  Movement 

NA 

1 

2 

3 

4 

5 

6 

7 
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#  Knowledge  and  skills: 

19.  Understanding  of  Battle  Space  Geometry 


Your  current  level 


poor 

Excellent 


NA 

1 

2 

3 

4 

5 

6 

13)  If  you  did  not  have  enough  time  to  become  familiar  with  the  infantry  TTPs  of  the  offense  or  defense 
(whichever  you  participated  in),  how  much  more  time  do  you  believe  you  would  need? 


14)  How  satisfied  were  you  with  your  overall  performance  while  using  the  CCM  TDS: 


1  =  Not 
Satisfied 


2 


3 


4  =Average 


5 


6 


7  =  Extremely 


Thanks  very  much  for  participating! 


NOTE: 
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Pilot  Study  Survey 


-  TDS  FINAL  - 


Please  fill  in  the  following  questionnaire.  Answer  all  questions  as  objectively  as  you  can. 

All  information  will  be  held  confidential. 

If  you  need  to  expand  any  answer  please  use  the  reverse  side  of  your  paper  sheet. 

1)  Which  of  the  following  items/drills  did  you  exercise  at  least  once  during  the  TDS  sessions?  (Circle  all  that 
apply  from  the  list  below.) 


# 

Knowledge  and  skills: 

1. 

Familiarity  with  a  five  paragraph  order 

2. 

Employ  Machine  guns  in  support  of  Offensive  Operations 

3. 

Employ  Machine  guns  in  support  of  Defensive  Operations 

4. 

Infantry  Offensive  Operations  in  Field 

5. 

Infantry  Defensive  Operations  in  Field 

6. 

Tactical  Movement  On  Objective 

7. 

Fire  and  Maneuver 

8. 

Infantry  Attack  in  Urban  Terrain 

9. 

Infantry  Defense  in  Urban  Terrain 

10. 

Issue  an  Order 

11. 

Conduct  Tactical  Movement  in  the  Offense 

2)  Of  the  actions  that  you  chose  in  the  previous  question  (Question  1),  with  which  did  you  have  the  most 
difficulty? 

(circle  three) 


# 

Knowledge  and  skills: 

1. 

Familiarity  with  a  five  paragraph  order 

2. 

Employ  Machine  guns  in  support  of  Offensive  Operations 

3. 

Employ  Machine  guns  in  support  of  Defensive  Operations 

4. 

Infantry  Offensive  Operations  in  Field 

5. 

Infantry  Defensive  Operations  in  Field 

6. 

Tactical  Movement  On  Objective 
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# 

Knowledge  and  skills: 

7. 

Fire  and  Maneuver 

8. 

Infantry  Attack  in  Urban  Terrain 

9. 

Infantry  Defense  in  Urban  Terrain 

10. 

Issue  an  Order 

11. 

Conduct  Tactical  Movement  in  the  Offense 

3)  Of  the  actions  that  you  chose  in  the  Question  1 ,  which  do  you  feel  you  improved  upon  most  during  the  TDG 

sessions?  (circle  three) 


# 

Knowledge  and  skills: 

1. 

Familiarity  with  a  five  paragraph  order 

2. 

Employ  Machine  guns  in  support  of  Offensive  Operations 

3. 

Employ  Machine  guns  in  support  of  Defensive  Operations 

4. 

Infantry  Offensive  Operations  in  Field 

5. 

Infantry  Defensive  Operations  in  Field 

6. 

Tactical  Movement  On  Objective 

7. 

Fire  and  Maneuver 

8. 

Infantry  Attack  in  Urban  Terrain 

9. 

Infantry  Defense  in  Urban  Terrain 

10. 

Issue  an  Order 

11. 

Conduct  Tactical  Movement  in  the  Offense 

4)  When  you  think  about  your  own  knowledge  of  infantry  tactics,  techniques,  and  procedures,  how  would  you 
rank  them?  (please  circle  NA  or  one  number  between  1  and  7  in  each  line) 


# 

Knowledge  and  skills: 

Poor 

Excellent 

Your  current  level 

1. 

Familiarity  with  a  five  paragraph  order 

NA 

1 

2 

3 

4 

5 

6 

7 

2. 

Employ  Machine  guns  in  support  of  Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3. 

Employ  Machine  guns  in  support  of  Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4. 

Infantry  Offensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5. 

Infantry  Defensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6. 

Tactical  Movement  On  Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7. 

Fire  and  Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 
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# 

Knowledge  and  skills: 

Your  current  level 

Poor 

Excellent 

8. 

Infantry  Attack  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9. 

Infantry  Defense  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

10. 

Issue  an  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

11. 

Conduct  Tactical  Movement  in  the  Offense 

NA 

1 

2 

3 

4 

5 

6 

7 

5)  When  you  think  about  your  own  knowledge  of  infantry  tactics,  techniques,  and  procedures,  how  much  did 
you  improve  in  each?  (please  circle  NA  or  one  number  between  1  and  7  in  each  line) 


# 

Knowledge  and  skills: 

They 

Improved 

Improve 

greatly 

Your  improvement 

did 

at 

not 

all 

1 

Familiarity  with  a  five  paragraph  order 

NA 

1 

2 

3 

4 

5 

6 

7 

2 

Employ  Machine  guns  in  support  of  Offensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

3 

Employ  Machine  guns  in  support  of  Defensive 
Operations 

NA 

1 

2 

3 

4 

5 

6 

7 

4 

Infantry  Offensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

5 

Infantry  Defensive  Operations  in  Field 

NA 

1 

2 

3 

4 

5 

6 

7 

6 

Tactical  Movement  On  Objective 

NA 

1 

2 

3 

4 

5 

6 

7 

7 

Fire  and  Maneuver 

NA 

1 

2 

3 

4 

5 

6 

7 

8 

Infantry  Attack  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

9 

Infantry  Defense  in  Urban  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

1 

Issue  an  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

1 

Conduct  Tactical  Movement  in  the  Offense 

NA 

1 

2 

3 

4 

5 

6 

7 

6)  How  would  you  rank  your  ability  to  navigate  through  the  virtual  world  and  use  the  controls  in  simulation? 


1. 

NAVIGATION/MOVING 
THROUGH  THE  TERRAIN: 

1  = 

POOR 

2 

3 

4  = AVERAGE 

5 

6 

7  = 

EXCELLENT 

2. 

OTHER  CONTROLS: 

1  = 

POOR 

2 

3 

4  = AVERAGE 

5 

6 

7  = 

EXCELLENT 

7)  Rate  the  simulation  ability  to  provide  you  with  good  visual  and  audio  cues  so  that  you  can  identify  all  threats 
during  the  mission: 


1. 

Visual  cues: 

1  =  poor 

2 

3 

4 

=  average 

5 

6 

7 

=  excellent 

2. 

Audio  cues: 

1  =  poor 

2 

3 

4 

=  average 

5 

6 

7 

=  excellent 
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8)  Rate  the  following  statements  (circle  one  number  for  each  statement): 


1. 

“During  the  session  with  computer  simulation,  I 
felt  like  I  was  taking  part  in  training.” 

1  =  strongly 
disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 

2. 

“During  the  session  with  computer  simulation,  I 
felt  like  my  actions  in  virtual  world  had  no 
consequences  to  the  others  and  to  the  virtual 
world.” 

1  =  strongly 
disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 

3. 

“During  the  session  with  computer  simulation,  I 
felt  like  I  was  part  of  the  group  working  together.” 

1  =  strongly 
disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 

4. 

“During  the  session  with  computer  simulation,  I 
felt  like  I  was  playing  a  game.” 

1  =  strongly 
disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 

5. 

“During  the  session  with  computer  simulation,  I 
felt  isolated  from  the  others.” 

1  =  strongly 
disagree 

2 

3 

4  = 

neutral 

5 

6 

7  = 

strongly 

agree 

9)  In  your  opinion  what  was  the  overall  quality  of  training  in  computer  based  training  simulation?  (circle  one) 


1  =  not  satisfactory  2  3  4  =  average  5  6  7  =  excellent 

10)  Throughout  all  the  TDS  sessions,  please  rate  the  quality  of  your  tactical  decisions,  (circle  one  number) 


1  =  poor  2  3 

4  =  satisfactory 

5 

6 

7  =  excellent 

11)  What  were  your  expectations  regarding  the  ability  of  simulation  to  train  you  in  the  previous  set  of  skills 

mentioned  in  Questions  1,  2,  and  3? 

1  =  low  2  3 

4  =  average 

5 

6 

7  =high 

12)  How  well  did  the  use  of  simulation  meet  your  expectations  regarding  training  value? 


1  =  not  satisfactorily 

2 

3 

4  =  satisfactorily 

5 

6 

7  =  exceptionally 

well 

13)  Circle  one  number  that  best  reflects  your  opinion: 


When  you  think  about  your  future  training 

1  =  not 

2 

3 

4  = 

5 

6 

7  =  very 

needs,  to  what  extent  would  you  like  it  to  be 

at  all 

moderately 

much 

supplemented  with  computer  based 

simulations  like  CCM? 

Thank  you  very  much  for  participating  in  the  study. 
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APPENDIX  J  -  TDG/TDS  SCHEDULE  (EXPERIMENT  2) 


TDG 

-  APPROXIMATED  TBS  Design  - 


Phase  I  -  Pre-Training  (Individual) 

SETTING:  Classroom/Lab  (Watkins  212A  &  MOCAP  Lab) 

METHOD  OF  INSTRUCTION:  TDG 

00-10  TIME:  10  Minutes  -  Initial  Long  Pre-Questionnaire 

SCENARIO:  1-la-  Town(a) 

1 0-30  TIME:  20  Minutes  -  Conduct  of  the  TDG 

TIME:  Minutes  -  What  is  a  TDG? 

-Planning  Sequence 

1 .  Read  and  Understand  Operation  Order 

2.  Estimate  Enemy  activity 

3.  Plan  your  response 

4.  What-if  your  plan. 

5.  Move  troops  to  attack/defense  positions 

6.  Conduct  Offensive/Defensive  operations 

7.  Conduct  After- Action  Review 

-Offensive  Tactics  Training  (OPFOR) 

1.  Support  by  Fire  Positions 

2.  Distribution  of  Forces 

3.  Exposure  (use  terrain  to  mask  movement  and  positions) 

4.  Use  Smoke  to  cover  movement. 

5.  Create  and  Use  a  Fire  Support  Plan 

6.  Don’t  give  conflicting  orders  (ex.  “Dig  in”  and  “Move  Fast”) 

7.  Maneuver  Warfare 

a)  Fire  and  Maneuver  [SAM-K  (Suppress,  Assess,  Maneuver,  Kill)]  (I  cover  you,  you  cover  me  during  movements). 

b)  De-escalation  of  the  Fire  support  upon  approach 

i.  Artillery 

ii.  BN  mortars 

iii.  CO  mortars 

iv.  Medium  machine  guns 

v.  Squad  automatic  weapons 

vi.  M-16 

vii.  Grenades 

8.  Tactical  movement  (moving  fast  or  moving  covert) 

9.  Long/Unsupported  movements  are  bad 

10.  Use  of  the  reserve 

-  Defensive  Tactics  Training  (MARINES) 

1 .  Dig  in  when  possible 

2.  Envision  Enemy’s  attack  based  on  the  OpOrd 

3.  Select  the  ground  where  you  can  stop  the  enemy  attack 

4.  Select  troop  battle  positions  based  on  your  expectation 

5.  Ensure  interlocking  Sectors  of  Fire  on  this  ground 

6.  Envision  the  enemy’s  reaction  to  this  initial  contact 

7.  Have  a  plan  to  counter  the  enemy’s  reaction  to  initial  contact 

8.  Give  key  weapons  primary  and  secondary  positions  of  expected  enemy  action/reaction 

9.  Use  Rifle  Squads  to  cover  the  flanks  of  your  key  weapons 

10.  Use  covered  movements  between  primary  and  supplemental  positions  (some  units  may  not  move) 

11.  Control  of  Fires 

12.  Be  flexible,  (i.e.,  Don’t  stick  to  your  plan  if  it  is  not  working.) 

13.  Use  primary,  alternate,  supplemental  positions,  a  reserve. 

NOTE:  Platoons  and  squads  are  not  large  enough  to  designate  a  static  reserve.  One  unit  should  be  designated  to  break  contact 
and  act  as  the  reserve  if  needed. 

14.  Establish  a  withdrawal  plan  if  you  need  to  evacuate  the  area,  (route,  sequence, 
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30-40  Conducting  a  10  minute  TDG  introduction 
00-02  Division  Of  The  Teams  Into  2  Groups 
02-04  Passing  out  The  Material  For  TDG 
04-10  Explanation  of  what  Each  Person  Is  Supposed  To 
40-45  TIME:  5  Minutes  -  After  Action 
45-50  TIME:  5  Minutes  -  Post  Questionnaire 

Phase  II  -  Tactics  Training  -  Defense 
(Team) 

SCENARIO:  3-la  -  Town  (b) 


00-05  TIME:  5  Minutes  Instructor  Introduction  with  Reading  of  OpOrder 

05-15  TIME:  10  Minutes 

TIME:  5  Minutes  Group  Discussion 

TIME:  5  Minutes  Individual  Student  Planning 
-Mission  Enemy  Terrain  Troops  -  Time  (METT-T) 

-conduct  map  recon 
-estimate  of  situation 
-sketch  tactical  plan  on  map 

15-25  TIME:  10  Minutes  Internal  Team  discussion  on  WHY  each  individual  chose  his  COA 

25-49  TIME:  24  Minutes  Team  Leader  Picked  and  Briefs  Overall  Plan  (Instructors  may  pause  for  critical  discussion  points) 

TIME  12:  Team  1  Briefs 
Time  12:  Team  2  Briefs 

49-54  TIME:  5  Minutes  -  Individuals  Write  down  the  good/bad  points  of  your  team’s  and  the  other  team’s  plan 
54-59  TIME:  5  Minutes  -  Post-Questionnaire  with  After  Action  Section 

(Subject  Matter  Experts  (SMEs)  Provide  Evaluation  on  Tactical  Decision-Making  and 
Platoon  Level  Leadership) 

59-65  TIME:  6  Minutes  -  Phase  II  After- Action  Review 


Source  of  Time  Information:  J.D.  DeForest  combined  with  DVD  of  Experiment  with  Dr.  Sadagic 

Phase  III  -  Tactics  Training  -  Offense 
(Team  Internal  Role  Shift) 

SCENARIO:  3-la-  Town  (b) 


00-05  TIME:  5  Minutes  Instructor  Introduction  with  Reading  of  OpOrder 

05-15  TIME:  10  Minutes 

TIME:  5  Minutes  Group  Discussion 

TIME:  5  Minutes  Individual  Student  Planning 
-Mission  Enemy  Terrain  Troops  -  Time  (METT-T) 

-conduct  map  recon 
-estimate  of  situation 
-sketch  tactical  plan  on  map 

15-25  TIME:  10  Minutes  Internal  Team  discussion  on  WHY  each  individual  chose  his  COA 

25-49  TIME:  24  Minutes  Team  Leader  Picked  and  Briefs  Overall  Plan  (Instructors  may  pause  for  critical  discussion  points) 

TIME  12:  Team  1  Briefs 
Time  12:  Team  2  Briefs 

49-54  TIME:  5  Minutes  -  Individuals  Write  down  the  good/bad  points  of  your  team’s  and  the  other  team’s  plan 

54-59  TIME:  5  Minutes  -  Post-Questionnaire  with  After  Action  Section 

(Subject  Matter  Experts  (SMEs)  Provide  Evaluation  on  Tactical  Decision-Making  and 
Platoon  Level  Leadership) 

59-65  TIME:  6  Minutes  -  Phase  II  After- Action  Review 

Source  of  Time  Information:  J.D.  DeForest  combined  with  DVD  of  Experiment  with  Dr.  Sadagic 

Phase  IV  -  Tactics  Training  -  Defense 
(Team  Internal  Role  Shift) 

SCENARIO:  2-la  -  Airfield 


00-05  TIME:  5  Minutes  Instructor  Introduction  with  Reading  of  OpOrder 
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05-15  TIME:  10  Minutes 

TIME:  5  Minutes  Group  Discussion 

TIME:  5  Minutes  Individual  Student  Planning 
-Mission  Enemy  Terrain  Troops  -  Time  (METT-T) 

-conduct  map  recon 
-estimate  of  situation 
-sketch  tactical  plan  on  map 

15-25  TIME:  10  Minutes  Internal  Team  discussion  on  WHY  each  individual  chose  his  COA 

25-49  TIME:  24  Minutes  Team  Leader  Picked  and  Briefs  Overall  Plan  (Instructors  may  pause  for  critical  discussion  points) 

TIME  12:  Team  1  Briefs 
Time  12:  Team  2  Briefs 

49-54  TIME:  5  Minutes  -  Individuals  Write  down  the  good/bad  points  of  your  team’s  and  the  other  team’s  plan 

54-59  TIME:  5  Minutes  -  Post-Questionnaire  with  After  Action  Section 

(Subject  Matter  Experts  (SMEs)  Provide  Evaluation  on  Tactical  Decision-Making  and 
Platoon  Level  Leadership) 

59-65  TIME:  6  Minutes  -  Phase  II  After- Action  Review 


Source  of  Time  Information:  J.D.  DeForest  combined  with  DVD  of  Experiment  with  Dr.  Sadagic 

Phase  V  -  Tactics  Training/Force  on  Force 
(Team  Internal  Role  Shift) 

SCENARIO:  6-la  -  Farm 

00-05  TIME:  5  Minutes  Instructor  Introduction  with  Reading  of  OpOrder 

05-15  TIME:  10  Minutes 

TIME:  5  Minutes  Group  Discussion 

TIME:  5  Minutes  Individual  Student  Planning 
-Mission  Enemy  Terrain  Troops  -  Time  (METT-T) 

-conduct  map  recon 
-estimate  of  situation 
-sketch  tactical  plan  on  map 

15-25  TIME:  10  Minutes  Internal  Team  discussion  on  WHY  each  individual  chose  his  COA 

25-49  TIME:  24  Minutes  Team  Leader  Picked  and  Briefs  Overall  Plan  (Instructors  may  pause  for  critical  discussion  points) 

TIME  12:  Team  1  Briefs 
Time  12:  Team  2  Briefs 

49-54  TIME:  5  Minutes  -  Individuals  Write  down  the  good/bad  points  of  your  team’s  and  the  other  team’s  plan 

54-59  TIME:  5  Minutes  -  Post-Questionnaire  with  After  Action  Section 

(Subject  Matter  Experts  (SMEs)  Provide  Evaluation  on  Tactical  Decision-Making  and 
Platoon  Level  Leadership) 

59-65  TIME:  6  Minutes  -  Phase  II  After- Action  Review 


Source  of  Time  Information:  J.D.  DeForest  combined  with  DVD  of  Experiment  with  Dr.  Sadagic 
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TDS 

-  Pilot  Study  Design  - 


Phase  I  -  Pre-Training 
(Individual  Intro  to  Tactics) 

SETTING:  Classroom/Lab  (Watkins  212A) 

METHOD  OF  INSTRUCTION:  Lecture  with  PowerPoint  Presentation.  CCM. 

00-10  TIME:  10  Minutes  -  Initial  Long  Pre-Questionnaire 
10-15  TIME:  5  Minutes  -  Instruction  via  Lecture  &  PPT. 

-Show  Opening  CCM  Screen 

-Describe  the  three  entry  points  (Command,  Editor,  Boot  Camp) 

-Go  into  Command 

-Operation  of  the  simulation 

1 .  Left  Click  =  Highlight  Unit,  Right  Click  =  Issue  Orders 

2.  Ambush  30m/Defend  200m 

3.  Firing/Range  indicator 

4.  #  on  Buildings  =  #  indicates  Levels 

5.  Navigation  -  arrows/map 

6.  To  get  Elevation  Data  -  Right  Click  on  terrain 

7.  Space  Bar  -  Leadership  Circle 

15-35  TIME:  20  Minutes  (10  Minutes  OPFOR/IO  Minutes  MARINES) 

-Planning  Sequence 

1 .  Read  and  Understand  Operation  Order 

2.  Estimate  Enemy  activity 

3.  Plan  your  response 

4.  What-if  your  plan. 

5.  Move  troops  to  attack/defense  positions 

6.  Conduct  Offensive/Defensive  operations 

7.  Conduct  After- Action  Review 

-Offensive  Tactics  Training  (OPFOR) 

1.  Support  by  Fire  Positions 

2.  Distribution  of  Forces 

3.  Exposure  (use  terrain  to  mask  movement  and  positions) 

4.  Use  Smoke  to  cover  movement. 

5.  Create  and  Use  a  Fire  Support  Plan 

6.  Don’t  give  conflicting  orders  (ex.  “Dig  in”  and  “Move  Fast”) 

7.  Maneuver  Warfare 

a)  Fire  and  Maneuver  [SAM-K  (Suppress,  Assess,  Maneuver,  Kill)]  (I  cover  you,  you  cover  me  during  movements). 

b)  De-escalation  of  the  Fire  support  upon  approach 

i.  Artillery 

ii.  BN  mortars 

iii.  CO  mortars 

iv.  Medium  machine  guns 

v.  Squad  automatic  weapons 

vi.  M-16 

vii.  Grenades 

8.  Tactical  movement  (moving  fast  or  moving  covert) 

9.  Long/Unsupported  movements  are  bad 

10.  Use  of  the  reserve 

-  Defensive  Tactics  Training  (MARINES) 

1 .  Dig  in  when  possible 

2.  Envision  Enemy’s  attack  based  on  the  OpOrd 

3.  Select  the  ground  where  you  can  stop  the  enemy  attack 

4.  Select  troop  battle  positions  based  on  your  expectation 

5.  Ensure  interlocking  Sectors  of  Fire  on  this  ground 

6.  Envision  the  enemy’s  reaction  to  this  initial  contact 

7.  Have  a  plan  to  counter  the  enemy’s  reaction  to  initial  contact 
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8.  Give  key  weapons  primary  and  secondary  positions  of  expected  enemy  action/reaction 

9.  Use  Rifle  Squads  to  cover  the  flanks  of  your  key  weapons 

10.  Use  covered  movements  between  primary  and  supplemental  positions  (some  units  may  not  move) 

11.  Control  of  Fires 

12.  Be  flexible,  (i.e.,  Don’t  stick  to  your  plan  if  it  is  not  working.) 

13.  Use  primary,  alternate,  supplemental  positions,  a  reserve. 

NOTE:  Platoons  and  squads  are  not  large  enough  to  designate  a  static  reserve.  One  unit  should  be  designated  to  break  contact  and  act 
as  the  reserve  if  needed. 

14.  Establish  a  withdrawal  plan  if  you  need  to  evacuate  the  area,  (route,  sequence, 

SCENARIO:  1-la  -  Town  (a)  as  OPFOR  for  Offense/Marines  for  Defense 
(Researchers  Available  for  Questions/Help) 

Individual  Hands-On  Session 
35-37  TIME:  2  Minutes  -  Placement  of  Troops 
37-47  TIME:  10  Minutes  -  Offensive  Training 
47-57  TIME:  10  Minutes  -  Defensive  Training 
57-62  TIME:  5  Minutes  -  After  Action 
62-67  TIME:  5  Minutes  -  Post  Questionnaire 


Phase  II  -  Team  Tactics 
(Team  Offense  or  Team  Defense) 

SETTING:  Classroom/Lab  (Watkins  212A  &  MOCAP  Lab) 

METHOD  OF  INSTRUCTION:  Lecture  with  CCM. 

00-10  TIME:  10  Minutes  -  Tactics  Review  from  Phase  I 
-  REVIEW  Offensive  Tactics  Training  (5  Minutes) 

-  REVIEW  Defensive  Tactics  Training  (5  Minutes) 

SCENARIO:  3-lb  -  Town  (b) 

(OPFOR  for  Offense/Marines  for  Defense  (Researchers  Available  for  Questions/Help)) 

10-12  TIME:  2  Minutes  -  Issue  Operations  Order 
12-15  TIME:  3  Minute  -  Planning 
15-17  TIME:  2  Minutes  -  Troops  Placement 
17-39  TIME:  22  Minutes  -  Team  CCM  Hands-On 

32-34  TIME:  2  Minutes  -  GRADE  Evaluation  of  Situational  Awareness 

39-44  TIME:  5  Minutes  -  Post-Questionnaire  with  After  Action  Section 

(Subject  Matter  Experts  (SMEs)  Provide  Evaluation  on  Tactical  Decision-Making  and 
Platoon  Level  Leadership) 

44-49  TIME:  5  Minutes  -  Phase  II  After- Action  Review 


Phase  III  -  Team  Tactics 

(Team  Offense  or  Team  Defense... Switch  from  Phase  II) 

SETTING:  Classroom/Lab  (Watkins  212A  &  MOCAP  Lab) 

METHOD  OF  INSTRUCTION:  CCM. 


SCENARIO:  3-lb  -  Town  (b) 

(OPFOR  for  Offense/Marines) 

05-07  TIME:  2  Minutes  -  Issue  Operations  Order 
07-10  TIME:  3  Minute  -  Planning 
10-12  TIME:  2  Minutes  -  Troops  Placement 
12-56  TIME:  44  Minutes  -  Team  CCM  Hands-On 
32-34  TIME:  2  Minutes  -GRADE  1 
44-46  TIME:  2  Minutes  -GRADE  2 
56-61  TIME:  5  Minutes  -  Post-Questionnaire  with  After  Action  Section 

(Subject  Matter  Experts  (SMEs)  Provide  Evaluation  on  Tactical  Decision-Making  and 
Platoon  Level  Leadership) 

61-66  TIME:  5  Minutes  -  Phase  III  After- Action  Review 
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Phase  IV  -  Team  Tactics 

(Team  Offense  or  Team  Defense... Switch  from  Phase  III) 

SETTING:  Classroom/Lab  (Watkins  212 A  &  MOCAP  Lab) 

METHOD  OF  INSTRUCTION:  CCM. 

SCENARIO:  2-lb  -  Airfield 

(OPFOR  for  Offense/Marines  for  Defense  (Researchers  Available  for  Questions/Help) 

05-07  TIME:  2  Minutes  -  Issue  Operations  Order 
07-10  TIME:  3  Minute  -  Planning 
10-12  TIME:  2  Minutes  -  Troops  Placement 
12-56  TIME:  44  Minutes  -  Team  CCM  Hands-On 
32-34  TIME:  2  Minutes  -  GRADE  1 
44-46  TIME:  2  Minutes  -  GRADE  2 
56-61  TIME:  5  Minutes  -  Post-Questionnaire  with  After  Action  Section 

(Subject  Matter  Experts  (SMEs)  Provide  Evaluation  on  Tactical  Decision-Making  and 
Platoon  Level  Leadership) 

61-66  TIME:  5  Minutes  -  Phase  IV  After- Action  Review 


Phase  V  -  Team  Tactics 

(Team  Offense  or  Team  Defense... Switch  from  Phase  IV) 


SETTING:  Classroom/Lab  (Watkins  212A  &  MOCAP  Lab) 
METHOD  OF  INSTRUCTION:  CCM. 


SCENARIO:  6-lb  -  Farm 

(OPFOR  for  Offense/Marines  for  Defense  (Researchers  Available  for  Questions/Help) 

05-07  TIME:  2  Minutes  -  Issue  Operations  Order 
07-10  TIME:  3  Minute  -  Planning 
10-12  TIME:  2  Minutes  -  Troops  Placement 
12-56  TIME:  44  Minutes  -  Team  CCM  Hands-On 
32-34  TIME:  2  Minutes  -  GRADE  1 
44-46  TIME:  2  Minutes  -  GRADE  2 
56-61  TIME:  5  Minutes  -  Post-Questionnaire  with  After  Action  Section 

(Subject  Matter  Experts  (SMEs)  Provide  Evaluation  on  Tactical  Decision-Making  and 
Platoon  Level  Leadership) 

61-66  TIME:  5  Minutes  -  Phase  V  After- Action  Review 
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APPENDIX  K  -  CCM  OPERATION  ORDERS  (EXPERIMENT  2) 


SESSION  1 


Module  1  Fight  1-la  OpOrder  (Marines  -  DEFENSE) 

I.  Situation:  Enemy  patrols,  mostly  squad  size,  have  become  increasingly  bolder  as  we 
have  withdrawn  (South).  I  think  we  can  expect  tentative  contact  here  in  the  village  this 
morning.  Battalion  withdraws  (South)  and  repositions  in  order  to  turn  over  this  sector  to 
allied  forces. 

II.  Mission:  “A”  Company  guards  the  Battalion  rear  in  order  to  prevent  Enemy 
interference  with  our  withdrawal. 

III.  Execution:  (1)  Platoon  is  detached  to  Battalion.  (1)  platoon  guards  the  town  while  (1) 
Platoon  moves  (South)  to  establish  next  rear  guard  Position. 

Fire  Support:  No  Close  Air  Support.  No  Artillery.  No  Mortars. 

Tasks: 

1st  Platoon:  Main  Effort.  Guard  the  Company  rear  in  order  to  prevent  Enemy 
interference  with  our  withdrawal. 

2nd  Platoon:  Detached. 

3rd  Platoon:  Move  (South)  and  establish  next  rear  guard  Position  in  order  to 
permit  1  st  Platoon  to  withdraw. 

Mortars:  Priority  of  Fires  to  3rd  Platoon. 

IV.  Admin  and  Logistics:  Standard  Operating  Procedure. 

V.  Command  and  Signal:  Standard  Operating  Procedure. 
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SESSION  1 


Module  1  Fight  1-la  OpOrder  (Opposing  Force  (OP FOR)  -  OFFENSE) 

I.  Situation:  An  Enemy  squad  with  a  machine  gun  holds  the  village.  I  think  they  missed 
their  withdrawal  and  are  now  planning  on  fighting  to  the  end.  Battalion  secures  a 
Defensive  Position  6k  (East)  in  order  to  destroy  an  anticipated  Enemy  Counter-Attack. 

II.  Mission:  “E”  Company  clears  the  town  in  order  to  remove  Enemy  forces. 

III.  Execution:  While  (1)  Main  Effort  Platoon  clears  (South)  through  the  village,  (1) 
Platoon  screens  (East)  of  the  village.  (1)  Platoon  Reserve. 

Fire  Support:  No  Close  Air  support.  No  Artillery. 

Tasks: 

1st  Platoon:  Main  Effort.  Clear  (South)  through  the  village  IOT  remove  Enemy 
forces. 

2nd  Platoon:  Screen  (East)  of  the  village  in  order  to  identify  and  report  Enemy 
activities. 

3rd  Platoon:  Reserve.  Follow  in  Trace  of  1st  Platoon.  Be  prepared  to  Cover  Main 
Effort. 

Sniper  Team:  You  are  in  Direct  Support  of  the  Main  Effort. 

IV.  Admin  and  Logistics:  Standard  Operating  Procedure. 

V.  Command  and  Signal:  Standard  Operating  Procedure. 
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SESSION  2  &  3 


Module  5  Fight  5-la  OpOrder  (OP FOR) 

I.  Situation:  Enemy  patrols,  mostly  squad  size,  have  become  increasingly  bolder  as  we 
have  withdrawn  (West).  I  think  we  can  expect  tentative  contact  here  in  the  village  this 
morning.  Battalion  withdraws  (West)  and  repositions  in  order  to  conduct  a  counter  attack. 

II.  Mission  “E”  Company  guards  the  Battalion  rear  in  order  to  prevent  Enemy 
interference  with  our  withdrawal. 

III.  Execution  (1)  Platoon  is  detached  to  Battalion.  (1)  Platoon  guards  the  town  while  (1) 
Platoon  moves  (West)  to  establish  next  rear  guard  Position. 

FS:  No  Close  Air  Support.  No  artillery.  No  Mortars. 

Tasks: 

1st  Platoon:  Main  Effort.  Guard  the  Company  rear  in  order  to  prevent  Enemy  interference 
with  our  withdrawal. 

2nd  Platoon:  Detached. 

3rd  Platoon:  Move  (West)  and  establish  next  rear  guard  Position  in  order  to  permit  1st 
Platoon  to  Withdraw. 

Mortars:  Priority  of  Fire  to  3rd  Platoon. 

IV.  Admin  and  Logistics:  Standard  Operating  Procedure. 

V.  Command  and  Signal:  Standard  Operating  Procedure. 
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SESSION  2  &  3 


Module  5  Fight  5-la  QpQrder  (MARINES-OFFEN SE) 

I.  Situation:  An  Enemy  Squad  with  a  Machine  Gun  holds  the  bank  building,  the  last 
organized  resistance  in  the  village.  I  think  they  missed  their  withdrawal  and  are  now 
planning  on  fighting  to  the  end.  Battalion  secures  a  defensive  position  6  kilometers 
(West)  in  order  to  stop  an  anticipated  Enemy  counter  attack. 

II.  Mission:  “A”  Company  clears  the  town  in  order  to  free  the  road  from  Enemy  control. 

III.  Execution:  While  (1)  Platoon  clears  (South),  the  Main  effort  Platoon  clears  (West)  to 
the  Bank.  Reserve  Platoon  then  clears  the  rest  of  town. 

FS:  No  Close  Air  Support.  No  Artillery.  Mortars  to  2nd  Platoon. 

Tasks: 

1st  Platoon:  Main  Effort.  Clear  (West)  to  the  bank  in  order  to  free  the  road  from  Enemy 
control. 

2nd  Platoon:  Clear  (South)  across  the  river  in  order  to  free  the  road  from  Enemy  control. 

3rd  Platoon:  RESERVE.  Be  Prepared  To  clear  the  town  in  order  to  free  the  road  from 
Enemy  control. 

Mortars:  Priority  of  Fire  is  to  2nd  Plat. 

IV.  Admin  and  Logistics:  Standard  Operating  Procedure. 

V.  Command  and  Signal:  Standard  Operating  Procedure. 
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SESSION  4 


Module  2  Fight  2-la  OpOrder  (Marines  -  OFFENSE) 

I.  Situation:  Scattered  Enemy  units,  maybe  (20)  soldiers,  remain  in  the  vicinity  of  the 
airfield.  They  may  have  a  Heavy  Machine  Gun,  mortar,  or  Anti-Aircraft  missile  that 
could  threaten  our  helicopters.  The  remainder  of  the  battalion  will  fly  into  the  airfield 
once  the  Anti-Aircraft  threat  is  reduced  in  order  to  stage  for  follow-on  operations.  I  think 
the  Enemy  will  avoid  contact  with  us  and  wait  to  target  the  helicopters. 

II.  Mission:  “A”  Company  clears  the  area  around  the  airfield  in  order  to  prevent  Enemy 
interference  with  air  operations. 

III.  Execution:  While  (1)  Platoon  blocks  the  (North)  access  road,  (2)  Platoons  clear  both 
sides  of  the  airfield  from  (North)  to  (South). 

FS:  Rotary  Wing  Close  Air  Support  On-Call.  No  Artillery.  Mortars  with  1st  Platoon. 
Tasks: 

1st  Platoon:  Main  Effort.  Clear  (West)  side  of  airfield  in  order  to  prevent  Enemy 
interference  with  air  operations. 

2nd  Platoon:  Clear  (East)  side  of  airfield  in  order  to  prevent  Enemy  interference 
with  air  operations. 

3rd  Platoon:  Block  (North)  approach  to  airfield  in  order  to  prevent  Enemy 
Counter-Attack. 

Mortars:  Priority  of  Fire  to  the  Main  Effort. 

IV.  Admin  and  Logistics:  Standard  Operating  Procedure. 

V.  Command  and  Signal:  Standard  Operating  Procedure. 
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_ SESSION  4 _ 

Module  2  Fight  2-la  OpOrder  (Opposing  Force  (OpFor)  -  DEFENSE) 

I.  Situation:  An  Enemy  unit,  possibly  Company  sized  is  located  10  kilometers  to  the 
(North  West).  They  may  be  intending  to  use  the  airfield  as  a  staging  area  for  FOF.  The 
remainder  of  our  Battalion  is  located  15  kilometers  to  the  (East).  I  think  the  Enemy  will 
attempt  to  seize  the  airfield  within  the  next  (2)  hours. 

II.  Mission:  “E”  Company  Controls  the  airfield  in  order  to  prevent  Enemy  use. 

III.  Execution:  (1)  Platoon  Clears  (West)  side  of  airfield,  (1)  Platoon  Clears  (East)  side  of 
airfield.  (1)  Platoon  detached  to  Battalion. 

FS:  No  Close  Air  Support.  No  Artillery.  Mortars  to  Main  Effort. 

Tasks: 

1st  Platoon:  Main  Effort.  Clear  the  (West)  side  of  airfield  in  order  to  prevent  Enemy  use. 
2nd  Platoon:  Clear  (East)  side  of  airfield  in  order  to  prevent  Enemy  use. 

3rd  Platoon:  Detached  Mortars:  Priority  of  Fire  to  Main  Effort. 

IV.  Admin  and  Logistics:  Standard  Operating  Procedure. 

V.  Command  and  Signal:  Standard  Operating  Procedure. 


156 


SESSION  5 


Module  6  Fight  6-la  OpOrder  (Marines  -  OFFENSE) 

I.  Situation:  An  Enemy  patrol  has  cut  the  road  at  the  village,  stranding  a  convoy  farther 
(East).  Enemy  is  believed  to  number  (12)  to  (15)  soldiers  and  have  shoulder- fired 
missiles.  The  convoy  reports  additional  Enemy  movement  in  the  hills  to  their  (East), 
trapping  them  in  place.  I  think  the  Enemy  will  focus  on  the  road  traffic  and  less  on  the 
overland  approaches  to  the  village.  This  main  road  links  our  Battalion,  4  kilometers 
(West),  with  2nd  Battalion,  17  kilometers  (East).  Tomorrow,  Battalion  moves  (North) 
and  secures  river  line  in  order  to  prevent  Enemy  crossings. 

II.  Mission:  “A”  Company  destroys  the  Enemy  in  the  village  in  order  to  clear  the  road  for 
convoy  operations. 

III.  Execution:  (2)  Platoons  envelop  the  village  from  the  (North).  Heavy  Machine  Gun 
Squad  provides  SBF  vicinity  of  the  road. 

FS:  Rotary  Wing  Close  Air  Support  On  call.  Artillery  On  call.  Mortars  with  Main 
Effort. 

Tasks: 

1st  Platoon:  Main  Effort.  Destroy  the  Enemy  in  the  village  in  order  to  clear  the  road  for 
convoy  operations. 

2nd  Platoon:  RESERVE.  Follow  in  trace  of  1st  Platoon  be  prepared  to  continue  Attack. 
3rd  Platoon:  Detached. 

Heavy  Machine  Gun  Squad:  Direct  Support  to  Main  Effort. 

Mortars:  Priority  of  Fire  to  Main  Effort. 

IV.  Admin  and  Logistics:  Standard  Operating  Procedure. 

V.  Command  and  Signal:  Standard  Operating  Procedure. 
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SESSION  5 


Module  6  Fight  6-la  OpOrder  (Opposing  Force  (OpFor)  -  DEFENSE) 

I.  Situation:  Enemy  Company  has  been  sighted  (West)  of  the  town.  I  think  the  Enemy 
will  attack  us  with  a  reinforced  platoon  here  in  town  in  order  to  gain  control  of  this  road 
network.  Battalion  secures  this  (East)-(West)  Main  Service  Route  in  order  to  prevent 
Enemy  interference  with  resupply  convoys. 

II.  Mission  “E”  Company  secures  the  town  in  order  to  protect  the  Main  Service  Route 
from  Enemy  interference. 

III.  Execution  (1)  Platoon  secures  the  town.  (1)  Platoon  secures  the  (South)  road.  (1) 
Platoon  is  detached. 

FS:  No  Close  Air  Support.  No  artillery.  No  Mortars. 

Tasks: 

1st  Platoon:  Main  Effort.  Secure  the  town  in  order  to  protect  the  Main  Service  Route. 

2nd  Platoon:  Detached. 

3rd  Platoon:  Move  (South)  and  secure  road  sector  in  order  to  protect  Main  Service  Route. 
Mortars:  Priority  of  Fire  is  to  3rd  Platoon. 

IV.  Admin  and  Logistics:  Standard  Operating  Procedure. 

V.  Command  and  Signal:  Standard  Operating  Procedure. 
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APPENDIX  L  -  TACTICS  MEMORY  SHEET  (EXPERIMENT  2) 
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Offensive  Tactics 


Use  Terrain  to  Mask  Mvmt  Use  of  the  Reserve 


Defensive  Tactics 


Dig  In  When  Possible 


Ensure  Interlocking 
Sectors  of  Fire 
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Defensive  Tactics 


Select  Killing  Ground 


Defensive  Tactics 


Select  Battle  Position 
Based  on  Expectation  of 
Enemy  Advance 


Use  Primary,  Alternate, 
Supplementary  Positions 

Primary 


Alternate 


Supplementary 
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APPENDIX  M  -  EXAMPLE  GRADE  SA  SHEET  (EXPERIMENT  2) 


Geographical  Recall  and  Analysis  of  Data  in  the  Environment  (GRADE) 


1e.  Choose  Location  on  Map 


2e.  Pick  Location  on  Map 


1b.  Pick  Location  on  Map 
ic.  Typo  Perceived  Unit 
S tee  (P-Platoonh  S-Squad,  T- 

1d. 

advance 


2b-  Pick  Location  on  Map 
2c -  Type  Perceived  Unit 
Slie  (i -Platoon,  S-Squad,  T- 
Team) 

2d.  Choose,  direction  of 
advance 

w 


2a.  Choose  Certainty 
Level  of  Friendly 
Location 
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APPENDIX  N  -  EVALUATION  FORM  (EXPERIMENT  2) 


TDS  -  Platoon  Leader  Leadership 

Evaluation  Sheet 

Subject  #: 

Team: 

Offense  /  Defense 

Evaluator: 

Date/Time: 

Likert  Scale  (1)  Poor  to  (7)  Excellent 

# 

Knowledge  and  skills: 

.  Platoon  Leader's  Current  Level 

Poor 

Excellentj 

PLANNER 

1 

Understood  Operations  Order 

NA 

1 

2 

3 

4 

5 

6 

7 

2 

Identified  Enemy  Most  Probably  Course  of  Action 
(MPCOA) 

NA 

1 

2 

3 

4 

5 

6 

■ 

3 

Created  a  Scheme  of  Maneuver  (SOM) 

NA 

1 

2 

3 

4 

5 

6 

a 

4 

Considers  Adjacent  Units 

NA 

1 

2 

3 

4 

5 

6 

a 

5 

Created  a  Fire  Support  Plan  (FSP) 

NA 

1 

2 

3 

4 

5 

6 

7 

6 

Identified  Secondary  MPCOA 

NA 

1 

2 

3 

4 

5 

6 

7 

7 

Considered  the  following  in  Mission  Planning 
Process 

NA 

1 

2 

3 

4 

5 

6 

■ 

8 

-  Mission 

NA 

1 

2 

3 

4 

5 

6 

7 

9 

-  Enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

10 

-  Troops  Available 

NA 

1 

2 

3 

4 

5 

6 

7 

11 

-  Terrain 

NA 

1 

2 

3 

4 

5 

6 

7 

12 

-  Time 

NA 

1 

2 

3 

mm 

5 

6 

7 

DECISION  MAKER 

13 

Identified  Enemy  of  Center  of  Gravity  and  Attacks 
It. 

NA 

1 

2 

3 

4 

5 

6 

7 

14 

Adjusted  to  Enemy  Situation 

NA 

1 

2 

3 

4 

5 

6 

7 

LEADER  /  COMMANDER 

15 

Took  Charge 

NA 

1 

2 

3 

4 

5 

6 

n 

16 

Maintained  Oversight  of  All  Subordinate  Units 

NA 

1 

2 

3 

4 

5 

6 

7 

COMMUNICATOR 

17 

Communicated  His  Plan/Intent  to  Subordinates 

NA 

1 

2 

3 

4 

5 

6 

7 

18 

Ensured  that  Subordinates  did  what  they  had  been 
told 

NA 

1 

2 

3 

4 

5 

6 

■ 

TACTICALLY  CUNNING 

19 

Used  deception  and  creativity  to  outwit  the  enemy 

NA 

1 

2 

3 

4 

5 

6 

7 

20 

Deviated  from  doctrine  when  necessary 

NA 

1 

2 

3 

4 

5 

6 

n 

Source:  TBS’s  Five  Pillars  of  Leadership  (Evaluated  by  instructor  and  survey 
results) 
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APPENDIX  O  -  PRESENTATION  OF  WORK 


On  numerous  occasions,  we  had  the  opportunity  to  present  this  work  to  the  NPS 
Human  Factors  Focus  group  and  during  the  Modeling  Virtual  Environments  and 
Simulation  (MOVES)  Open  House.  Throughout  2006  and  2007  as  we  would  come  up 
with  a  new  idea  or  completed  a  component  of  this  thesis,  we  would  ask  for  some  time  in 
the  schedule  to  present  our  findings  to  the  Human  Factors  Focus  Group  at  NPS  MOVES 
Institute.  This  group  was  a  collection  of  students  and  professors  who  were  interested  in 
the  human  experience  within  virtual  environments.  We  received  many  suggestions  and 
constructive  comments  on  our  work.  Once  we  had  completed  Experiment  2,  we  had  the 
privilege  of  presenting  our  overall  findings  at  the  2007  MOVES  Open  House  in  July 
2007.  We  were  pleased  with  the  response  from  the  audience  and  were  happy  to  see  that 
our  work  had  been  of  interest  to  some  within  the  military  and  from  the  private  sector  as 
well. 
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